compass python package ====================== Author: Xylar Asay-Davis date: 2020/11/16 Summary ------- While the existing COMPASS infrastructure has served us well in providing a framework for setting up MPAS test cases and test suites, several shortcomings have emerged over the years. First, new users have not found the current system of creating XML files that are parsed into python scripts, namelists and streams files very intuitive or easy to modify. Second, the current scripts and XML files do not lend themselves to code reuse, leading to a cumbersome system of symlinked scripts and XML config files. Third, the only way that users currently have of modifying test cases is to edit namelists, streams files and run scripts for each step individually after the test case has been set up. Fourth and related, there is not a way for users to easily constrain or modify how many cores a given test case uses, making it hard to configure test cases in a way that is appropriate for multiple machines. Fifth and also related, COMPASS does not currently have a way to provide machine-specific paths and other information that could allow for better automation on supported machines. Sixth, the directory structure imposed by COMPASS (``mpas_core/test_group/resoltuion/test_case/step``) is too rigid for many applications. Finally, COMPASS is not well documented and the documentation that does exist is not very helpful either for new users or for new developers interested in creating new test cases. The proposed ``compass`` python package should address these challenges with the hope of making the MPAS test cases significantly easier to develop and run. Requirements ------------ .. _req_easy: Requirement: Make test cases easy to understand, modify and create ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/12/04 Contributors: Xylar Asay-Davis, Luke Van Roekel Currently, test cases are written primarily in XML files that are then used to generate a python script along with namelist and streams files for MPAS-Model. We have found that this system is not very intuitive for new users or very easy to get started with. New users would likely have an easier time if test cases were written in a more direct way, using a common language rather than custom XML tags. Importantly, creating a test case should also be as easy as possible. There is a need to balance readability and reusability. There is a risk that the compass redesign, as it becomes heavily pythonic, may make it difficult for developers to contribute. But we can't go too far the other way either. We want the best balance possible between readability and reusibility. .. _req_shared_code: Requirement: Shared code ^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/11/16 Contributors: Xylar Asay-Davis Currently, there are two approaches to sharing code between COMPASS test cases. In some cases, shared code is part of an external package, often ``mpas_tools``, which is appropriate for code that may be used outside of COMPASS. However, this approach is cumbersome for testing, so it is not the preferred approach for COMPASS-specific code. In other cases, scripts are symlinked in test cases and run with test-case-specific flags. This approach is also cumbersome and does not lend itself to code reuse between scripts. Finally, many test cases attempt to share XML files using symlinks, a practice that has led to frequent unintended consequences when a linked file is modified with changes appropriate for only one of the test cases that uses it. A more sophisticate method for code reuse should be developed beyond symlinks to isolated scripts and shared XML files. .. _req_shared_options: Requirement: Shared configuration options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/11/16 Contributors: Xylar Asay-Davis Currently, COMPASS reads a configuration file as part of setting up test cases, but these configuration options are largely unavailable to test cases themselves. Some steps of some test cases (e.g. ``files_for_e3sm`` in some ``ocean/global_ocean`` test cases) have their own dedicated config files, but these are again separate from the config file used to set up test cases, are awkward to modify (requiring editing after test case generation). .. _req_core_count: Requirement: Ability specify/modify core counts ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/11/16 Contributors: Xylar Asay-Davis Some test cases involve multiple steps of running the MPAS model, each with a hard-coded number of cores (and often with a corresponding hard-coded number of PIO tasks), which makes it tedious to modify the number of cores or nodes that a given test case uses. This problem is exacerbated in test suites, where it is even more difficult and tedious to modify processor counts for individual test cases. A system is needed where the user can more easily override the default number of cores used in one or more steps of a test case. The number of PIO tasks and the stride between them should be updated automatically to accommodate the new core count. .. _req_machine_data: Requirement: Machine-specific data ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/11/16 Contributors: Xylar Asay-Davis Currently, many COMPASS test cases have hard-coded processor counts and related information that are likely only appropriate for one machine. Users must specify the paths to shared datasets such as meshes and initial conditions. Users must also know where to load the ``compass`` conda environment appropriate for running test cases. If information were available on the system being used, such as the number of cores per node and the locations of shared paths, test cases and the COMPASS infrastructure could take advantage of this to automate many aspects of setting up and running test cases that are currently unnecessarily redundant and tedious. .. _req_dir_struct: Requirement: Looser, more flexible directory structure ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/11/16 Contributors: Xylar Asay-Davis The directory structure currently imposed by COMPASS (``mpas_core/test_group/resoltuion/test_case/step``) is too rigid for many applications. Some test cases (e.g. convergence tests) require multiple resolutions within the test case. Some test groups would prefer to sort test cases based on another parameter or property besides resolution. It would be convenient if the directory structure could be more flexible, depending on the needs of a given test group and test case. Even so, it is important that the subdirectory of each test case and step is unique, they do not overwrite one another. .. _req_docs: Requirement: User- and developer-friendly documentation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/11/16 Contributors: Xylar Asay-Davis We need a set of user-friendly documentation on how to setup and activate an appropriate conda environment; build the appropriate MPAS core; list and setup a test case; and run the test case in via a batch queuing system. Similarly, we need a set of developer-friendly documentation to describe how to create a new "test group" with one or more "test cases", each made up of one or more "steps". .. _req_parallel: Requirement: Considerations related to running test cases in parallel ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/12/10 Contributors: Xylar Asay-Davis, Matt Hoffman In the longer term, we would like ot add the capability of running multiple test cases within a test suite in parallel with one another for reduced wall-clock time. Similarly, we would also like to support multiple steps within a test case running in parallel with one another (e.g. the forward runs with different viscosities in the baroclinic channel RPE test case). Full support for this capability will not be included in this design, but design choices should be mindful of this future addition in the hopes of minimizing future modifications, particularly to individual test cases. .. _req_res: Requirement: Resolution can be a test case parameter ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/12/04 Contributors: Xylar Asay-Davis, Mark Petersen Currently, resolution is hard-coded in the directory structure and in scripts for individual test groups like ``build_base_mesh.py``. This works for more complex meshes but for convergence tests, it is not useful to have a directory per resolution. Instead, it could be helpful to have a list of resolutions that can easily be altered (e.g. ``dx = {min, max, step}`` with a linear or log step) with either configuration options or within the code. For convergence tests, resolution is a parameter, rather than something fundamental. This could also reduce the number of test cases in the full list. .. _req_alter_code: Requirement: Test case code is easy to alter and rerun ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/12/04 Contributors: Xylar Asay-Davis, Mark Petersen In the current ``compass``, the created directories include soft links to scripts like ``build_base_mesh.py`` and ``add_initial_condition.py``. It is easy to edit that file and rerun it, and quickly iterate until one gets the desired result. New people also understand this workflow. The new design should still be easy to work with. .. _req_premade_ic: Requirement: Support for pre-made initial condition files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/12/04 Contributors: Xylar Asay-Davis, Mark Petersen Ideally, it should be possible for a given test case to either generate an initial condition or read a pre-made initial condition from a file (possibly downloading this file if it has not been cached). Alternatively, two different versions of a test case could exists, one with the generated and one with the pre-made initial condition. .. _req_batch: Requirement: Easy batch submission ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/12/04 Contributors: Xylar Asay-Davis, Mark Petersen There should be an easy way for users to submit batch jobs without having to create their own batch script or modify an example. Algorithm Design ---------------- .. _alg_easy: Algorithm design: Make test cases easy to understand, modify and and create ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis The proposed solution would be to write test cases as Python packages made up of modules, functions and classes within a larger ``compass`` package. A test case will descend from a base ``TestCase`` class with a constructor for adding steps to the test case (the equivalent of parsing ``config_driver.xml`` in the current implementation), a ``configure()`` method for adding test-case-specific config options, and a ``run()`` method to run the steps and perform validation. Each step of a test case (equivalent to the other ``config_*.xml`` files) will descend from the ``Step`` base class. Each step class will include a constructor to add input, output, namelist and streams files and collect other information on the step (equivalent to parsing ``config_*.xml``); a ``setup()`` method that downloads files, makes symlinks, creates namelist and streams files; and a ``run()`` method that runs the step. Steps may be shared between test cases. A balance will have to be struck between code reusability and readability within each test group (a set of test cases). Readability would be improved by using Jinja2 templates for code generation, rather than via string manipulation within python scripts as is the case in the current COMPASS implementation: .. code-block:: python #!/usr/bin/env python import pickle import configparser from mpas_tools.logging import LoggingContext def main(): with open('test_case_{{ test_case.name }}.pickle', 'rb') as handle: test_case = pickle.load(handle) test_case.steps_to_run = ['{{ step.name }}'] test_case.new_step_log_file = False with open('{{ step.name }}.pickle', 'rb') as handle: step = pickle.load(handle) config = configparser.ConfigParser( interpolation=configparser.ExtendedInterpolation()) config.read('{{ step.config_filename }}') test_case.config = config # start logging to stdout/stderr test_name = step.path.replace('/', '_') with LoggingContext(name=test_name) as logger: test_case.logger = logger test_case.run() if __name__ == '__main__': main() A Jinja2 template uses curly braces (e.g. ``{{ test_case.name }}``) to indicate where an element of the template will be replaced by a python variable or dictionary value. In this example, ``{{ test_case.name }}`` will be replaced with the contents of ``test_case['name']`` in the python code, and similarly for other replacements in the template. Other than the replacements, the code can be read as normal, in contrast to the existing approach of python scripts that define other python scripts via a series of string formatting statements. The only XML files that would be used would be templates for streams files, written in the same syntax as the resulting streams files. .. code-block:: xml Templates for namelist files would have the same basic syntax as the resulting namelist files: .. code-block:: ini config_write_output_on_startup = .false. config_run_duration = '0000_00:15:00' config_use_mom_del2 = .true. config_implicit_bottom_drag_coeff = 1.0e-2 config_use_cvmix_background = .true. config_cvmix_background_diffusion = 0.0 config_cvmix_background_viscosity = 1.0e-4 Regarding the balance between reusability and readability, it is difficult to generalize this to the whole redesign. To some degree this will be a choice left to each test case. It will be difficult to reuse code across test cases and steps within a test group without some degree of increased complexity. The redesign will attempt to include simpler examples, perhaps with less code reuse, that can serve as starting points for the creation of new test cases. These "prototype" test cases will include additional documentation and commenting to help new developers follow them and use them to design their own test cases. Even without the compass redesign, a certain familiarity with use of python packages is somewhere between recommended and required to add new test cases to COMPASS. With the redesign, it will become essentially inevitable that developers have a certain minimum level of familiarity with python. While there may be a learning curve, it is hoped that these skills will pay off far beyond COMPASS in a way that learning the existing XML-based approach cannot be. .. _alg_shared_code: Algorithm design: Shared code ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis By organizing both the test cases themselves and shared framework code into a ``compass`` Python package, code reuse and organization should be greatly simplified. The organization of the package will be as follows: .. code-block:: none - compass/ - / - .cfg - .py - / - tests/ - / - / - .py - .cfg - namelist. - streams. - .py - .py - .cfg - namelist. - streams. - .py - / The proposed solution would slightly modify the naming conventions currently used in COMPASS. An MPAS core (descended from the ``MpasCore`` base class) would be the same as it now is -- corresponding to an MPAS dynamical core such as ``ocean`` or ``landice``. A test group (descended from ``TestGroup``) would would be the equivalent of a "configuration" in legacy COMPASS -- a group of test cases such as ``global_ocean`` or ``MISMIP3D``. For at least two reasons described in :ref:`req_dir_struct`, we do not include ``resolution`` as the next level of hierarchy. Instead, a test group contains test cases (descended from ``TestCase``), which can be given any convenient name and relative path to distinguish it from other test cases within that test group. Several variants of a test case can define by varying a parameter or other characteristic (including resolution) but there need not be separate packages or modules for each. This is an important aspect of the code reuse provided by this approach. Each test case is made up of several steps (e.g. ``base_mesh``, ``initial_state``, ``forward``). Legacy COMPASS' documentation referred to a test case as a "test" and a step as a "case", but users have found this naming convention to be confusing so the proposed solution tries to make a clearer distinction between a test case and a step within a test case. In addition to defining test cases and steps, MPAS cores and test groups can also include "framework" python code that could be more general (e.g. for creating meshes or initial conditions). The main ``compass`` package would also include several framework modules and package, some for infrastructure related to listing, setting up and cleaning up test cases, and others for tasks common to many test cases. The methods of the base classes, particularly of the ``Step`` class, are also an important part of the framework that can be used to indicate what the input and output files for the step are and how to create the namelist and streams files. Here is an example of a step that that is defined using a combination of methods from ``Step`` (e.g. ``self.add_input_file()``) and framework functions (e.g. ``run_model()``): .. code-block:: python from compass.model import run_model from compass.step import Step class Forward(Step): """ A step for performing forward MPAS-Ocean runs as part of baroclinic channel test cases. Attributes ---------- resolution : str The resolution of the test case """ def __init__(self, test_case, resolution, name='forward', subdir=None, cores=1, min_cores=None, threads=1, nu=None): """ Create a new test case Parameters ---------- test_case : compass.TestCase The test case this step belongs to resolution : str The resolution of the test case name : str the name of the test case subdir : str, optional the subdirectory for the step. The default is ``name`` cores : int, optional the number of cores the step would ideally use. If fewer cores are available on the system, the step will run on all available cores as long as this is not below ``min_cores`` min_cores : int, optional the number of cores the step requires. If the system has fewer than this number of cores, the step will fail threads : int, optional the number of threads the step will use nu : float, optional the viscosity (if different from the default for the test group) """ self.resolution = resolution if min_cores is None: min_cores = cores super().__init__(test_case=test_case, name=name, subdir=subdir, cores=cores, min_cores=min_cores, threads=threads) self.add_namelist_file('compass.ocean.tests.baroclinic_channel', 'namelist.forward') self.add_namelist_file('compass.ocean.tests.baroclinic_channel', 'namelist.{}.forward'.format(resolution)) if nu is not None: # update the viscosity to the requested value options = {'config_mom_del2': '{}'.format(nu)} self.add_namelist_options(options) self.add_streams_file('compass.ocean.tests.baroclinic_channel', 'streams.forward') self.add_input_file(filename='init.nc', target='../initial_state/ocean.nc') self.add_input_file(filename='graph.info', target='../initial_state/culled_graph.info') self.add_output_file(filename='output.nc') def setup(self): """ Set up the test case in the work directory, including downloading any dependencies """ self.add_model_as_input() def run(self): """ Run this step of the test case """ run_model(self) .. _alg_shared_config: Algorithm design: Shared configuration options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis In the work directory, each test case will have a single config file that is populated during the setup phase and which is symlinked within each step of the test case. The idea of having a single config file per test case, rather than one for each step, is to make it easier for users to modify config options in one place at runtime before running all the steps in a test case. This will hopefully avoid the tedium of altering redundant namelist or config options in each step. The config files will be populated from default config options provided in several config files within the ``compass`` package. Any config options read in from a later config file will override the same option from an earlier config file, so the order in which the files are loaded is important. The proposed loading order is: * A top level default config file related downloading files and partitioning meshes for parallel execution * machine config file (found in ``compass/machines/.cfg``, with ``default`` being the machine name if none is specified) * MPAS core config file (found in ``compass//.cfg``) * test group config file (found in ``compass//tests//.cfg``) * any additions or modifications made within the test case's ``configure()`` method. * the config file passed in by the user at the command line (if any). The ``configure()`` method allows each test case to load one or more config files specific to the test case (e.g. ``.cfg`` within the test case's package) and would also allow calls to ``config.set()`` that define config options directly. The resulting config file would be written to ``.cfg`` within the test case directory and symlinked to each step subdirectory as stated above. Algorithm design: Ability specify/modify core counts ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis Each step will specify the "target" number of cores, the minimum possible number of cores, a number of treads, the maximum memory it will be allowed to use, and the maximum amount of disk space it can use. These specifications are with the ``WorkerQueue`` approach in mind for future parallelism, as explained in :ref:`alg_parallel`. The total number of available cores will be determined via python or slurm commands. An error will be raised if too few cores are available for a particular step. Otherwise, the step will run on the minimum of the target number of cores or the total available. Some test cases (e.g. those within the ``global_ocean`` test group) will allow the user to specify the target and minimum number of cores as config options, meaning they can be set to non-default values before running the test case. Config options are common to all steps within a test case, but the target and minimum cores are a property of each step that must be known before it is run (again for reasons related to a likely strategy for future parallelism in :ref:`alg_parallel`). This means that a test case will need to parse the config options and use them to determine the number of cores each step needs to run with as part of its ``run()`` method before calling ``super().run()`` from the base class to run the steps. Parsing config options and updating the target and minimum cores in a step will need to happen in each test cases that supports this capability. From there, shared infrastructure will take care of determining if sufficient cores are available and how many to run each step with if so. Developers of individual test cases will not need to worry about this. Here is an example from the ``Init`` test case from the ``GlobalOcean`` test group: .. code-block:: python def run(self): """ Run each step of the testcase """ config = self.config steps = self.steps_to_run if 'initial_state' in steps: step = self.steps['initial_state'] # get the these properties from the config options step.cores = config.getint('global_ocean', 'init_cores') step.min_cores = config.getint('global_ocean', 'init_min_cores') step.threads = config.getint('global_ocean', 'init_threads') if 'ssh_adjustment' in steps: step = self.steps['ssh_adjustment'] # get the these properties from the config options step.cores = config.getint('global_ocean', 'forward_cores') step.min_cores = config.getint('global_ocean', 'forward_min_cores') step.threads = config.getint('global_ocean', 'forward_threads') # run the steps super().run() ... Shared infrastructure can also be used to set the number of PIO tasks to one per node, using the number of cores for a given step and the number of cores per node from the machine config file (see :ref_`alg_machine_data`). .. _alg_machine: Algorithm design: Machine-specific data ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis The machine config file mentioned in :ref:`alg_shared_config` would have the following config options: .. code-block:: cfg # The paths section describes paths that are used within the ocean core test # cases. [paths] # The root to a location where the mesh_database, initial_condition_database, # and bathymetry_database for MPAS-Ocean will be cached ocean_database_root = /usr/projects/regionalclimate/COMMON_MPAS/ocean/grids/ # The root to a location where the mesh_database and initial_condition_database # for MALI will be cached landice_database_root = /usr/projects/regionalclimate/COMMON_MPAS/mpas_standalonedata/mpas-albany-landice # the path to the base conda environment where compass environments have # been created compass_envs = /usr/projects/climate/SHARED_CLIMATE/anaconda_envs/base # The parallel section describes options related to running tests in parallel [parallel] # parallel system of execution: slurm or single_node system = slurm # whether to use mpirun or srun to run the model parallel_executable = srun # cores per node on the machine cores_per_node = 36 # the slurm account account = e3sm # the number of multiprocessing or dask threads to use threads = 18 The various ``paths`` would help with finding mesh or initial condition files. The database root paths depend on the MPAS core, so new paths would need to be added for new cores. A strategy for setting environment variables, activating the appropriate conda environment, and loading compiler and MPI modules for each machine will be explored as a follow-up project and is not part of this design. The ``parallel`` options are intended to contain all of the machine-specific information needed to determine how many cores a given step would require. The use of python thread parallelism will not be part of the first version of the ``compass`` package described in this design document but is expected to be incorporated in the coming year. An appropriate value for ``threads`` for each machine will likely need determined as that capability gets more exploration but is left as a placeholder for the time being. .. _alg_dir_struct: Algorithm design: Looser, more flexible directory structure ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis Each test case and step will be defined by a unique subdirectory within the work directory. Within the base work directory, the first two levels of subdirectories will be conceptually the same as in the current implementation: ``mpas_core/test_group``. However, test cases will be free to determine the (unique) subdirectory structure beyond this top-most level. Many existing test cases will likely stick with the ``resolution/test_case/step`` organization structure imposed in the legacy COMPASS framework, but others may choose a different way of organizing (and, indeed, many test cases already have given the ``resolution`` subdirectory a name that is seemingly unrelated to the mesh resolution). A unique subdirectory for each test case and step will be provided as the ``subdir`` argument to the base class's constructor (i.e. ``super().__init__()`` or will be taken from the ``name`` argument if ``subdir`` is not provided. .. code-block:: python name = 'restart_test' self.resolution = resolution subdir = '{}/{}'.format(resolution, name) super().__init__(test_group=test_group, name=name, subdir=subdir) COMPASS will list test cases based on their full paths within the work directory, since this is the way that they can be uniquely identified. .. _alg_docs: Algorithm design: User- and developer-friendly documentation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2020/04/13 Contributors: Xylar Asay-Davis Documentation using ``sphinx`` and the ``ReadTheDocs`` template will be built out in a manner similar to what has already been done for: * `geometric_features `_ * `pyremap `_ * `MPAS-Tools `_ * `MPAS-Analysis `_ The documentation will include: * A user's guide for * setting up the conda environment * listing, setting up, and cleaning up test case * regression suites * creating and modifying config files * more details on each MPAS core, test group, test case and step * machine-specific instructions * A developer's guide: * A quick start * An overview (e.g. the design philosophy) * A section for each MPAS core * A subsection describing the test groups * A sub-subsection for each test case and its steps * A subsection for the MPAS core's framework code * A description of the ``compass`` framework code: * for use within test cases * for listing, setting up and cleaning up test cases * for managing regression test suites * An automated documentation of the API pulled from docstrings Eventually, but probably not as part of the current design, the documentation will also include: * A developer's guide for creating new test cases * MPAS-core-specific details for developing new test cases * More detailed tutorials: * Running a test case * Running the regression suite .. _alg_parallel: Algorithm design: Considerations related to running test cases in parallel ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis I plan to use `parsl `_ to support parallelism between both test cases and steps within a test case. After reading documentation, running tutorials, and beginning prototyping, it seems that the relatively new `WorkQueueExecutor `_ is likely to be the approach within Parsl that allows the level of flexibility and control that we would need. However, this is a new enough feature that it is still considered to be "beta" and is not available in the latest release (v1.0.0). So it seems premature to settle on this design choice or to begin to incorporate it into code (except perhaps as a separate prototype). Even so, some design choices can be made with future support for Parsl in mind. Each step of a test case will be required to provide full paths to its input and output files so that, in the future, Parsl can determine dependencies between test cases and their steps using these files and control execution accordingly. This will be the only method for determining dependencies, so steps will have to be accurate in providing their inputs and outputs to avoid errors, race conditions, or unnecessary blocking. Test cases with an test suite and steps within a test case will also need to be ordered in such a way that outputs of a "prerequisite" step are always defined before the inputs of any subsequent steps that need them as inputs. In the future, this should allow ``compass`` to associate each input file with a so-called Parsl ``DataFuture``, which will allow each step of a test case to run only when all of its input files are available. Also with Parsl in mind, the ``Step`` base class includes a specified maximum memory and disk usage. Currently, these are set to an arbitrary reference value of 1GB each but will be calibrated to the actual approximate usage of each step once this can be determined using debugging output from Parsl. This design solution will be fleshed out further in a separate document at a later date. .. _alg_res: Algorithm design: Resolution can be a test case parameter ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis As mentioned in :ref:`alg_shared_code` and :ref:`alg_dir_struct`, resolution will no longer be part of the directory structure for test cases and no restrictions will be placed on how individual test cases handle resolution or mesh generation. To facilitate shared code, a test group can use the same code for a step that generates a mesh and/or initial condition for different resolutions, e.g. passing in the resolution or mesh name as an argument to the step's constructor. .. _alg_alter_code: Algorithm design: Test case code is easy to alter and rerun ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis When ``python -m compass setup`` or ``python -m compass suite`` is run from a local ``compass`` repo as opposed to the conda package, the package creates a local symlink within each test case and step's work directory to the ``compass`` package. A developer can edit any files within the package either using the symlink or in the original local repo and then simply rerun the test case or step without having to rerun setup. Changes do not require a test build of a conda package or anything like that. After some discussion about adding symlinks to individual python files within the ``compass`` package, it was decided that this has too many risks of being misunderstood, having unintended consequences, and could be difficult to implement. .. _alg_premade_ic: Algorithm design: Support for pre-made initial condition files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis, Mark Petersen To a large degree, the implementation of this requirement will be left up to individual test cases. It should not be difficult to add a config option to a given test case selecting whether to generate an initial condition or read it from a file (and skipping initialization steps if the latter). The suggested approach would be to put an initial condition in the ``initial_condition_database`` under a directory structure similar to the ``compass`` work directory. The initial condition would have a date stamp so new initial conditions could be added over time without breaking backwards compatibility. However, this work will be considered outside the scope of this design document and is only discussed to ensure that the proposed design does not hinder a future effort in this direction. .. _alg_batch: Algorithm design: Easy batch submission ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/01/14 Contributors: Xylar Asay-Davis, Mark Petersen Rather than having users create their own batch scripts from scratch, a simper solution would be to generate a job script appropriate for a given machine from a template. This has been done for performance tests, `see example `_ for single line command. An alternative will be to use ``parsl`` to handle the SLURM (or other) submission. Prototyping that is currently underway will help to decide which approach we use for individual test cases. ``parsl`` will most likely be used for test suites. This work will not be part of the current implementation but an effort will be made to ensure that the design doesn't hinder later automatic generation of batch scripts. Additional information such as a default account name could be added to machine-specific config files to aid in this process. Implementation -------------- The implementation of this design can be found in the branch: `xylar/compass/compass_1.0 `_ and on the pull request at: https://github.com/MPAS-Dev/compass/pull/28 .. _imp_easy: Implementation: Make test cases easy to understand, modify and and create ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis As already discussed, this requirement is somewhat in conflict with :ref:`req_shared_code`, in that shared code within a test case tends to lead to a bit more complexity but considerably less redundancy. In addition to the constructor (``__init__()``), the ``TestCase`` base class has 2 other methods, ``configure()`` and ``run()``, that child classes are expected to override to set config options and perform additional tasks beyond just running the steps that belong to the test case. Similarly, in addition to the constructor, the ``Step`` base class has 2 other methods, ``setup()`` and ``run()`` for setting up and running the step. Each of these is described below. constructors ~~~~~~~~~~~~ When test cases and steps are instantiated, the constructor method ``__init__()`` is called. I will not go into the details of what happens in the ``TestCase`` and ``Step`` base classes when this happens because the idea is that developers of new test cases would not need to know these details. The constructors always need to take the parent (a ``TestGroup`` or ``TestCase`` object, respectively) and they can have additional arguments (such as the resolution or other parameters). The constructors must always call the base class' constructor ``super().__init__()`` with, at a minimum, the parent and the name of the test case or step as arguments. As an example, here is the constructor for the ``Default`` test case in the ``BaroclinicChannel`` test group in the ``Ocean`` MPAS core: .. code-block:: python class Default(TestCase): """ The default test case for the baroclinic channel test group simply creates the mesh and initial condition, then performs a short forward run on 4 cores. Attributes ---------- resolution : str The resolution of the test case """ def __init__(self, test_group, resolution): """ Create the test case Parameters ---------- test_group : compass.ocean.tests.baroclinic_channel.BaroclinicChannel The test group that this test case belongs to resolution : str The resolution of the test case """ name = 'default' self.resolution = resolution subdir = '{}/{}'.format(resolution, name) super().__init__(test_group=test_group, name=name, subdir=subdir) self.add_step( InitialState(test_case=self, resolution=resolution)) self.add_step( Forward(test_case=self, cores=4, threads=1, resolution=resolution)) And here is the constructor of the ``InitialState`` step: .. code-block:: python class InitialState(Step): """ A step for creating a mesh and initial condition for baroclinic channel test cases Attributes ---------- resolution : str The resolution of the test case """ def __init__(self, test_case, resolution): """ Update the dictionary of step properties Parameters ---------- test_case : compass.TestCase The test case this step belongs to resolution : str The resolution of the test case """ super().__init__(test_case=test_case, name='initial_state') self.resolution = resolution for file in ['base_mesh.nc', 'culled_mesh.nc', 'culled_graph.info', 'ocean.nc']: self.add_output_file(file) In this case, the argument ``resolution`` is passed in when the test case is created, and is passed on to the step when it is created within the test case's constructor. Both the test case and the step save the resolution in an attribute ``self.resolution`` of the class. The developer of a test case can add any number of parameters as attributes of each class in this way for later use in the test case or step. For example, the ``Default`` test case later uses the resolution to call a shared ``configure()`` function: .. code-block:: python def configure(self): """ Modify the configuration options for this test case. """ baroclinic_channel.configure(self.resolution, self.config) The shared function uses the resolution to determine other config options: .. code-block:: python def configure(resolution, config): """ Modify the configuration options for one of the baroclinic test cases Parameters ---------- resolution : str The resolution of the test case config : configparser.ConfigParser Configuration options for this test case """ res_params = {'10km': {'nx': 16, 'ny': 50, 'dc': 10e3}, '4km': {'nx': 40, 'ny': 126, 'dc': 4e3}, '1km': {'nx': 160, 'ny': 500, 'dc': 1e3}} if resolution not in res_params: raise ValueError('Unsupported resolution {}. Supported values are: ' '{}'.format(resolution, list(res_params))) res_params = res_params[resolution] for param in res_params: config.set('baroclinic_channel', param, '{}'.format(res_params[param])) Since all MPAS cores, test groups, test cases and steps are constructed as part of listing, setting up, and cleaning up test cases and test suites, it is important that these methods only perform a minimum of work to describe the test case and should not directly download or read files, or perform any complex computations. Because of these considerations, the ``Step`` base class includes infrastructure for identifying input and output files, and creating a "recipe" for setting up namelist and streams files within ``__init__()`` without actually downloading files, creating symlinks, parsing templates, or writing files. A step is allowed to: * call ``self.add_input_file()`` indicate files that should be symlinked from the ``compass`` package or from another step (in this or another test case) * call ``self.add_input_file()`` indicate files that should be downloaded from the LCRC server (or elsewhere) * call ``self.add_output_file()`` to indicate output files that will be produced by running the step * call ``self.add_namelist_file()`` or ``self.add_namelist_options()`` to add to the "recipe" for updating namelist options * call ``self.add_streams_file()`` to update the "recipe" for defining streams file during setup These functions can also be called on the step from the test case's constructor, e.g. ``step.add_namelist_file()``. This might be convenient when adding namelist options that are specific to the test case when you are using the same step class for many test cases. Namelist options always begin with a template produced when the MPAS model is compiled. Replacements are stored as keys and values in a python dictionary. For convenience, they can be read from easy-to-read files similar to the namelist files themselves but without sections: .. code-block:: none config_time_integrator = 'split_explicit' config_dt = '02:00:00' config_btr_dt = '00:06:00' config_run_duration = '0000_06:00:00' config_hmix_use_ref_cell_width = .true. config_write_output_on_startup = .false. config_use_debugTracers = .true. Such a file can be added within ``__init__()`` like this: .. code-block:: python class ForwardStep(Step): def __init__(self, test_case, mesh, init, time_integrator, name='forward', subdir=None, cores=None, min_cores=None, threads=None): ... self.add_namelist_file( 'compass.ocean.tests.global_ocean', 'namelist.forward') if mesh.with_ice_shelf_cavities: self.add_namelist_file( 'compass.ocean.tests.global_ocean', 'namelist.wisc') self.add_streams_file( 'compass.ocean.tests.global_ocean', 'streams.forward') The namelist recipe can be updated with multiple calls to ``self.add_namelist_file()`` as in this example, or it can be altered with a python dictionary of options by calling ``self.add_namelist_options()``. Streams files are in XML format and are therefore a little bit trickier to define. The recipe is always defined by adding a streams file with ``self.add_streams_file()`` as in the example above. A typical streams file might look like: .. code-block:: xml The file only has to provide attributes of a ```` or ```` tag if they differ from the defaults in the MPAS-model template. If ````, ```` and/or ```` tags are included in a stream, these will always replace the default contents of the stream. If none are provided, the default constants will be used. There is currently no mechanism for adding or removing ``vars``, etc. from a stream because that seemed to be a feature that was rarely used or found to be useful in the legacy COMPASS implementation. configure() ~~~~~~~~~~~ Test cases do not have very many options for customization. The main one is customizing the config file that is shared between all steps in the test case. The framework sets up a ``self.config`` attribute for each test case, and a test case can override the ``configure()`` method to modify these config options. One way to update ``config`` is by calling ``compass.config.add_config()`` to add options from a config file, typically found in the package (directory) for the test case: .. code-block:: python from compass.config import add_config from compass.io import symlink ... def configure(self): """ Modify the configuration options for this test case """ add_config(self.config, 'compass.landice.tests.enthalpy_benchmark.A', 'A.cfg', exception=True) with path('compass.landice.tests.enthalpy_benchmark', 'README') as \ target: symlink(str(target), '{}/README'.format(self.work_dir)) Another way is to call the ``config.set()`` method: .. code-block:: python def configure(self): """ Modify the configuration options for this test case """ # We want to visualize all test cases by default self.config.set('eismint2_viz', 'experiment', 'a, b, c, d, f, g') Config options in ``config`` will be written to a config file in the work directory called ``.cfg``, where ```` is the name of the test case. These config options differ from parameters (such as ``resolution`` in the example above) that are attributes of test case's class in that config options could be changed by a user before running the test case. Attributes of the test case are not available in a format where users could easily alter them and are unchanged between when the test case was set up and when it is run. Typically, config options that are specific to a test case will go into a config section with the same name as the test group. In the example above, we used a special section for visualization within the ``eismint2`` test group called ``eismint2_viz``. Developers can use whichever section name makes sense as long as the section names are different from those used by the framework such as ``[paths]`` and ``[parallel]``. It is also possible to create symlinks within ``configure()``, e.g. to a README file that applies to all steps in a test case, as shown above. Steps do not have a ``configure()`` method because they share the same ``config`` with the other steps in the test case. The idea is that it should be relatively easy to change config options for all the steps in the test case in one place. setup() ~~~~~~~ Test cases do not have a ``setup()`` method because the only setting up they typically include is to update config options in ``configure()``. The step may call ``self.add_input_file()``, ``self.add_output_file()``, `self.add_namelist_file()``, ``self.add_namelist_options()`` or ``self.add_streams_file()`` to add inputs, outputs, and update the recipes for namelist and streams files. Any operations that require explicit references to the work directory (i.e. ``self.work_dir``) or make use of config options (``self.config``) have to happen in ``setup()`` rather than ``__init__()`` because neither of these attributes are defined within ``__init__()``. Calls to ``self.add_model_as_input()``, which adds a symlink to the MPAS model's executable, must also happen in ``setup()`` because the path to the executable is a config option. run ~~~ The ``run()`` method of a test case should, at a minimum, call the base class' ``super().run()`` to run all the steps in the test case. It can also: * read config options and use them to update the number of cores and threads that a step can use * perform validation of variables and timers Here is a relatively complex example: .. code-block:: python from compass.validate import compare_variables ... def run(self): """ Run each step of the testcase """ step = self.mesh_step config = self.config # get the these properties from the config options step.cores = config.getint('global_ocean', 'mesh_cores') step.min_cores = config.getint('global_ocean', 'mesh_min_cores') # run the step super().run() variables = ['xCell', 'yCell', 'zCell'] compare_variables(variables, config, self.work_dir, filename1='mesh/culled_mesh.nc') The ``run()`` method of a step does the main "job" of the step so the contents will very much depend on the purpose of the step. Many steps will use Metis to split the domain across processors and then call the MPAS model, which can be done trivially with a call to ``run_model()``: .. code-block:: python from compass.model import run_model ... def run(self): """ Run this step of the testcase """ run_model(self) global ocean test group ~~~~~~~~~~~~~~~~~~~~~~~ The global ocean test group includes many other test cases and steps, and is quite complex compared to idealized test cases, so may need the most discussion. The ``global_ocean`` test group works with variable resolution meshes, requiring more significant numbers of parameters and even a function for defining the resolution. For this reason, it turned out to be more practical to define each mesh as its own python package: .. code-block:: none - compass/ - ocean/ - ocean.cfg - __init__.py - tests/ - global_ocean ... - mesh - ec30to60 - dynamic_adjustment - __init__.py - streams.template - __init__.py - ec30to60.cfg - namelist.split_explicit - qu240 - dynamic_adjustment - __init__.py - streams.template - __init__.py - namelist.rk4 - namelist.split_explicit - qu240.cfg ... The ``mesh`` module includes an intermediate step class ``MeshStep`` for defining meshes. ``MeshStep`` includes a method ``build_cell_width_lat_lon()`` that child classes must override to define the mesh resolution. To implement a new global mesh, one would need to define the resolution in the ``__init__.py`` file: .. code-block:: python import numpy as np from compass.ocean.tests.global_ocean.mesh.mesh import MeshStep class QU240Mesh(MeshStep): """ A step for creating QU240 and QUwISC240 meshes """ def __init__(self, test_case, mesh_name, with_ice_shelf_cavities): """ Create a new step Parameters ---------- test_case : compass.TestCase The test case this step belongs to mesh_name : str The name of the mesh with_ice_shelf_cavities : bool Whether the mesh includes ice-shelf cavities """ super().__init__(test_case, mesh_name, with_ice_shelf_cavities, package=self.__module__, mesh_config_filename='qu240.cfg') def build_cell_width_lat_lon(self): """ Create cell width array for this mesh on a regular latitude-longitude grid Returns ------- cellWidth : numpy.array m x n array of cell width in km lon : numpy.array longitude in degrees (length n and between -180 and 180) lat : numpy.array longitude in degrees (length m and between -90 and 90) """ dlon = 10. dlat = dlon constantCellWidth = 240. nlat = int(180/dlat) + 1 nlon = int(360/dlon) + 1 lat = np.linspace(-90., 90., nlat) lon = np.linspace(-180., 180., nlon) cellWidth = constantCellWidth * np.ones((lat.size, lon.size)) return cellWidth, lon, lat A developer would also need to define any namelist options for forward runs that are specific to this mesh (once for RK4 and once for split-explicit if both time integrators are supported): .. code-block:: none config_time_integrator = 'split_explicit' config_dt = '00:30:00' config_btr_dt = '00:01:00' config_run_duration = '0000_01:30:00' config_mom_del2 = 1000.0 config_mom_del4 = 1.2e11 config_hmix_scaleWithMesh = .true. config_use_GM = .true. The developer would define config options to do with the number of cores and vertical layers (both of which the user could change at runtime) as well as metadata to include in the output files: .. code-block:: cfg # Options related to the vertical grid [vertical_grid] # the type of vertical grid grid_type = 60layerPHC # options for global ocean testcases [global_ocean] ## config options related to the initial_state step # number of cores to use init_cores = 36 # minimum of cores, below which the step fails init_min_cores = 8 # maximum memory usage allowed (in MB) init_max_memory = 1000 # maximum disk usage allowed (in MB) init_max_disk = 1000 ## config options related to the forward steps # number of cores to use forward_cores = 128 # minimum of cores, below which the step fails forward_min_cores = 36 # maximum memory usage allowed (in MB) forward_max_memory = 1000 # maximum disk usage allowed (in MB) forward_max_disk = 1000 ## metadata related to the mesh # the prefix (e.g. QU, EC, WC, SO) prefix = EC # a description of the mesh and initial condition mesh_description = MPAS Eddy Closure mesh for E3SM version ${e3sm_version} with enhanced resolution around the equator (30 km), South pole (35 km), Greenland (${min_res} km), ${max_res}-km resolution at mid latitudes, and ${levels} vertical levels # E3SM version that the mesh is intended for e3sm_version = 2 # The revision number of the mesh, which should be incremented each time the # mesh is revised mesh_revision = 3 # the minimum (finest) resolution in the mesh min_res = 30 # the maximum (coarsest) resolution in the mesh, can be the same as min_res max_res = 60 # The URL of the pull request documenting the creation of the mesh pull_request = <<>> Finally, the developer would implement the ``dynamical_adjustment`` test case, using one of the existing spin-up test cases as a kind of a template. These test cases descend from the ``DynamicalAdjustment`` class, which itself descends from ``TestCase``. .. code-block:: python from compass.ocean.tests.global_ocean.dynamic_adjustment import \ DynamicAdjustment from compass.ocean.tests.global_ocean.forward import ForwardStep class QU240DynamicAdjustment(DynamicAdjustment): """ A test case performing dynamic adjustment (dissipating fast-moving waves) from an initial condition on the QU240 MPAS-Ocean mesh """ def __init__(self, test_group, mesh, init, time_integrator): """ Create the test case Parameters ---------- test_group : compass.ocean.test.global_ocean.GlobalOcean The global ocean test group that this test case belongs to mesh : compass.ocean.tests.global_ocean.mesh.Mesh The test case that produces the mesh for this run init : compass.ocean.tests.global_ocean.init.Init The test case that produces the initial condition for this run time_integrator : {'split_explicit', 'RK4'} The time integrator to use for the forward run """ restart_times = ['0001-01-02_00:00:00', '0001-01-03_00:00:00'] restart_filenames = [ 'restarts/rst.{}.nc'.format(restart_time.replace(':', '.')) for restart_time in restart_times] super().__init__(test_group=test_group, mesh=mesh, init=init, time_integrator=time_integrator, restart_filenames=restart_filenames) module = self.__module__ # first step step_name = 'damped_adjustment_1' step = ForwardStep(test_case=self, mesh=mesh, init=init, time_integrator=time_integrator, name=step_name, subdir=step_name) namelist_options = { 'config_run_duration': "'00-00-01_00:00:00'", 'config_Rayleigh_friction': '.true.', 'config_Rayleigh_damping_coeff': '1.0e-4'} step.add_namelist_options(namelist_options) stream_replacements = { 'output_interval': '00-00-01_00:00:00', 'restart_interval': '00-00-01_00:00:00'} step.add_streams_file(module, 'streams.template', template_replacements=stream_replacements) step.add_output_file(filename='../{}'.format(restart_filenames[0])) self.add_step(step) # final step step_name = 'simulation' step = ForwardStep(test_case=self, mesh=mesh, init=init, time_integrator=time_integrator, name=step_name, subdir=step_name) namelist_options = { 'config_run_duration': "'00-00-01_00:00:00'", 'config_do_restart': '.true.', 'config_start_time': "'{}'".format(restart_times[0])} step.add_namelist_options(namelist_options) stream_replacements = { 'output_interval': '00-00-01_00:00:00', 'restart_interval': '00-00-01_00:00:00'} step.add_streams_file(module, 'streams.template', template_replacements=stream_replacements) step.add_input_file(filename='../{}'.format(restart_filenames[0])) step.add_output_file(filename='../{}'.format(restart_filenames[1])) self.add_step(step) Whew! That was a lot, thanks for bearing with me. .. _imp_shared_code: Implementation: Shared code ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis The package includes myriad examples of code sharing so I will highlight a few. compass framework ~~~~~~~~~~~~~~~~~ The ``compass`` framework (classes, modules and packages not in the MPAS-core-specific packages) has a lot of code that is shared across existing test cases and could be very useful for future ones. Most of the framework currently has roughly the same functionality as legacy COMPASS, but it has been broken into more modules that make it clear what functionality each contains, e.g. ``compass.namelists`` and ``compass.streams`` are for manipulating namelists and streams files, respectively; ``compass.io`` has functionality for downloading files from LCRC and creating symlinks; and ``compass.validation`` can be used to ensure that variables are bit-for-bit identical between steps or when compared with a baseline, and to compare timers with a baseline. This functionality was all included in 4 very long scripts in legacy COMPASS. One example that doesn't have a clear analog in legacy COMPASS is the ``compass.parallel`` module. It contains two functions: ``get_available_cores_and_nodes()``, which can find out the number of total cores and nodes available for running steps. within an MPAS core ~~~~~~~~~~~~~~~~~~~ Legacy COMPASS shares functionality with an MPAS core by having scripts at the MPAS core level that are linked within test cases and which take command-line arguments that function roughly the same way as function arguments. But these scripts are not able to share any code between them unless it is from ``mpas_tools`` or another external package. Am MPAS core in ``compass`` could, theoretically, build out functionality as complex as in MPAS-Model if desired. Indeed, it is my ambition to gradually replace "init mode" in MPAS-Ocean with equivalent python functionality, starting with simpler test cases. This has already been accomplished for the 3 idealized ocean test cases included in the proposed design. The current shared functionality in the ``ocean`` MPAS core includes: * ``compass.ocean.namelists`` and ``compass.ocean.streams``: namelist replacements and streams that are similar to MPAS-core-level templates in legacy COMPASS. Current templates are for adjusting sea surface height in ice-shelf cavities, and outputting variables related to frazil and land-ice fluxes, * ``compass.ocean.suites``: the ocean test suites * ``compass.ocean.vertical``: supports for 1D vertical coordinates and the 3D z* coordinate. * ``compass.ocean.iceshelf``: computes sea-surface height and land-ice pressure, and adjusts them to match one another * ``compass.ocean.particles``: initialization of particles * ``compass.ocean.plot``: plots initial state and 1D vertical grid within a test group ~~~~~~~~~~~~~~~~~~~ So far, the most common type of shared code within test groups are modules defining steps that are used in multiple test cases. For example, the ``BaroclinicChannel`` test group uses shared modules to define the ``InitialState`` and ``Forward`` steps of each test case. Configurations also often include namelist and streams files with replacements to use across test cases. In addition to shared steps, the ``GlobalOcean`` test group includes some additional shared modules: * ``compass.ocean.tests.global_ocean.mesh``: defines properties of each global mesh (as well as a ``Mesh`` test case) * ``compass.ocean.tests.global_ocean.metadata``: determines the values of a set of metadata related to the E3SM mesh name, initial condition, conda environment, etc. that are added to nearly all ``global_ocean`` NetCDF output * ``compass.ocean.tests.global_ocean.subdir``: helps with maintaining the slightly complex subdirectory structure within ``global_ocean`` test cases. The shared code in ``global_ocean`` could easily define hundreds of different test cases using the QU240 (or QUwISC240) mesh. This is possible because the same conceptual test (e.g. restart) can be defined: * with or without ice-shelf cavities * with the PHC or EN4 1900 initial conditions * with or without BGC support * with the RK4 or split-explicit time integrators In practice, this is overkill and many of these variants will never be used so they are not currently made available. Also, I want to note that it is because of this flexibility that I added an RK4 restart test, which failed and showed us that there was a recent problem with RK4 restarts (https://github.com/MPAS-Dev/MPAS-Model/issues/777). within a test case ~~~~~~~~~~~~~~~~~~ There aren't too many cases so far where reuse of code within a test case is particularly useful. The main way this currently occurs is when the same module for a step gets used multiple times within a test case. For example, the `baroclinic_channel.rpe_test` test case uses the same forward run with 5 different viscosities. .. _imp_shared_config: Implementation: Shared configuration options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis As discussed in :ref:`alg_shared_config`, the proposed design builds up the config file for a given test from several sources. Some of the config options are related to setting up the test case (e.g. the locations of cached data files) but the majority are related to running the steps of the test case. During setup of a test case and its steps, The config file is assembled from a number of sources. Before the ``configure()`` method of the test case is called, config options come from: * the default config file, ``default.cfg``, which sets a few options related to downloading files during setup (whether to download and whether to check the size of files already downloaded) * the machine config file (using ``machines/default.cfg`` if none was specified) with information on the parallel system and (typically) the paths to cached data files * the MPAS core's config file. For the MPAS-Ocean core, this sets default paths to the MPAS model build (including the namelist templates). It uses "extended interpolation" in the config file to use config opitons within other config options, e.g. ``model = ${paths:mpas_model}/ocean_model``. * the test group's config file if one is found. For idealized test groups, these include config options that were previously init-mode namelist options. For ``global_ocean``, these include defaults for mesh metadata (again using "extended interpolation"); the default number of cores and other resource usage for mesh, init and forward steps; and options related to files created for E3SM initial conditions. Then, the ``configure()`` method is called on the test case itself. All of the current ocean test cases first call a shared ``configure()`` function at the test group level, e.g.: .. code-block:: python from compass.ocean.tests.global_ocean.configure import configure_global_ocean ... def configure(self): """ Modify the configuration options for this test case """ configure_global_ocean(test_case=self, mesh=self.mesh) where ``configure_global_ocean()`` is: .. code-block:: python from compass.config import add_config def configure_global_ocean(test_case, mesh, init=None): """ Modify the configuration options for this test case Parameters ---------- test_case : compass.TestCase The test case to configure mesh : compass.ocean.tests.global_ocean.mesh.Mesh The test case that produces the mesh for this run init : compass.ocean.tests.global_ocean.init.Init, optional The test case that produces the initial condition for this run """ config = test_case.config mesh_step = mesh.mesh_step add_config(config, mesh_step.package, mesh_step.mesh_config_filename, exception=True) if mesh.with_ice_shelf_cavities: config.set('global_ocean', 'prefix', '{}wISC'.format( config.get('global_ocean', 'prefix'))) add_config(config, test_case.__module__, '{}.cfg'.format(test_case.name), exception=False) # add a description of the initial condition if init is not None: initial_condition = init.initial_condition descriptions = {'PHC': 'Polar science center Hydrographic ' 'Climatology (PHC)', 'EN4_1900': "Met Office Hadley Centre's EN4 dataset from 1900"} config.set('global_ocean', 'init_description', descriptions[initial_condition]) # a description of the bathymetry config.set('global_ocean', 'bathy_description', 'Bathymetry is from GEBCO 2019, combined with BedMachine ' 'Antarctica around Antarctica.') if init is not None and init.with_bgc: # todo: this needs to be filled in! config.set('global_ocean', 'bgc_description', '<<>>') if mesh.with_ice_shelf_cavities: config.set('global_ocean', 'wisc_description', 'Includes cavities under the ice shelves around Antarctica') In this case, a config options related to the mesh are loaded, then those related to ice-shelf cavities (if they are included in the mesh), then those specific to the test case itself. Although none of the existing ocean test cases do so, further changes could be made to the config file beyond those at the test group level. Indeed, there is no reason a test case cannot just set its config options or read them from a file without calling a test-group-level function, this is just a convenience. Finally, config options are taken from the user's config file if one was passed in with the ``-f`` or ``--config_file`` commandline flag: .. code-block:: bash python -m compass setup -n 10 11 12 13 14 \ -w ~/scratch/mpas/test_baroclinic_channel -m anvil -f ocean.cfg python -m compass suite -s -c ocean -t nightly -m anvil -f ocean.cfg \ -w ~/scratch/mpas/test_nightly A typical config file resulting from all of this looks like: .. code-block:: cfg [download] download = True check_size = False verify = True [parallel] system = single_node parallel_executable = mpirun cores_per_node = 8 threads = 8 [paths] mpas_model = /home/xylar/code/mpas-work/compass/compass_1.0/MPAS-Model/ocean/develop mesh_database = /home/xylar/data/mpas/meshes initial_condition_database = /home/xylar/data/mpas/initial_conditions bathymetry_database = /home/xylar/data/mpas/bathymetry_database [namelists] forward = /home/xylar/code/mpas-work/compass/compass_1.0/MPAS-Model/ocean/develop/default_inputs/namelist.ocean.forward init = /home/xylar/code/mpas-work/compass/compass_1.0/MPAS-Model/ocean/develop/default_inputs/namelist.ocean.init [streams] forward = /home/xylar/code/mpas-work/compass/compass_1.0/MPAS-Model/ocean/develop/default_inputs/streams.ocean.forward init = /home/xylar/code/mpas-work/compass/compass_1.0/MPAS-Model/ocean/develop/default_inputs/streams.ocean.init [executables] model = /home/xylar/code/mpas-work/compass/compass_1.0/MPAS-Model/ocean/develop/ocean_model [ssh_adjustment] iterations = 10 [global_ocean] mesh_cores = 1 mesh_min_cores = 1 mesh_max_memory = 1000 mesh_max_disk = 1000 init_cores = 4 init_min_cores = 1 init_max_memory = 1000 init_max_disk = 1000 init_threads = 1 forward_cores = 4 forward_min_cores = 1 forward_threads = 1 forward_max_memory = 1000 forward_max_disk = 1000 add_metadata = True prefix = QU mesh_description = MPAS quasi-uniform mesh for E3SM version ${e3sm_version} at ${min_res}-km global resolution with ${levels} vertical level bathy_description = Bathymetry is from GEBCO 2019, combined with BedMachine Antarctica around Antarctica. init_description = <<>> e3sm_version = 2 mesh_revision = 1 min_res = 240 max_res = 240 max_depth = autodetect levels = autodetect creation_date = autodetect author = Xylar Asay-Davis email = xylar@lanl.gov pull_request = https://github.com/MPAS-Dev/compass/pull/28 [files_for_e3sm] enable_ocean_initial_condition = true enable_ocean_graph_partition = true enable_seaice_initial_condition = true enable_scrip = true enable_diagnostics_files = true comparisonlatresolution = 0.5 comparisonlonresolution = 0.5 comparisonantarcticstereowidth = 6000. comparisonantarcticstereoresolution = 10. comparisonarcticstereowidth = 6000. comparisonarcticstereoresolution = 10. [vertical_grid] grid_type = tanh_dz vert_levels = 16 bottom_depth = 3000.0 min_layer_thickness = 3.0 max_layer_thickness = 500.0 Unfortunately, all comments are lost in the process of combining config options. Comments are not parsed by ``ConfigParser``, and there is not a standard for which comments are associated with which options. So users would need to search through the code for the original config or look through the documentation to know what the config options are used for. In the future, we could consider implementing our own customized version of ``ConfigParser`` that preserves comments. Implementation: Ability specify/modify core counts ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/01/16 Contributors: Xylar Asay-Davis The ``Step`` class includes two attributes, ``cores`` and ``min_cores``, which should be set by the time the ``run()`` method gets called. ``cores`` is the target number of cores for the step and ``min_cores`` is the minimum number of cores, below which the test case would probably fail. Before a step is run, ``compass`` finds out how many total cores are available to run the test. If the number is below ``self.min_cores``, an error is raised. Otherwise, the test case will run with ``self.cores`` or the number of available cores, whichever is lower. The idea is that the same test case could be run efficiently on one or more nodes of an HPC machine but could also be run on a laptop or desktop if the minimum number of required cores is reasonable. There are a variety of ways that the ``cores`` and ``min_cores`` attributes can be set. The most straightforward is to set them by calling the base class' ``__init__()``. They could be passed through from calls to the child class' ``__init__()``: .. code-block:: python def __init__(self, test_case, cores=1, min_cores=None): """ Create a new test case Parameters ---------- test_case : compass.TestCase The test case this step belongs to cores : int, optional the number of cores the step would ideally use. If fewer cores are available on the system, the step will run on all available cores as long as this is not below ``min_cores`` min_cores : int, optional the number of cores the step requires. If the system has fewer than this number of cores, the step will fail """ if min_cores is None: min_cores = cores super().__init__(test_case=test_case, name='forward', cores=cores, min_cores=min_cores) or just hard coded: .. code-block:: python def __init__(self, test_case): """ Create a new test case Parameters ---------- test_case : compass.TestCase The test case this step belongs to """ if min_cores is None: min_cores = cores super().__init__(test_case=test_case, name='forward', cores=4, min_cores=1) Or they could be defined later in the process, at setup or in the test case's ``run()`` method. (Defining them in the step's ``run()`` is too late, since the number of cores to actually use is determined before this call is made.) In ``global_ocean``, the number of cores and minimum cores are set using config options. Since users could modify these before calling the ``run.py`` script, they are parsed in the test case's ``run()`` function before ``run_steps()`` is called: .. code-block:: python def run(self): """ Run each step of the testcase """ config = self.config # get the these properties from the config options for step_name in self.steps_to_run: step = self.steps[step_name] # get the these properties from the config options step.cores = config.getint('global_ocean', 'forward_cores') step.min_cores = config.getint('global_ocean', 'forward_min_cores') step.threads = config.getint('global_ocean', 'forward_threads') # run the steps super().run() The ``steps_to_run`` attribute of the test case is a list of the subset of the steps that were actually requested to run from the test case. For example, if you run a step on its own, it still actually runs the test case but only requesting that one step. Some test cases include steps that are not run by default, and this is specified by passing ``run_by_default=False`` as an argument to ``self.add_step()`` when adding the step in the test case's constructor: .. code-block:: python def __init__(self, test_group, mesh_type): """ Create the test case Parameters ---------- test_group : compass.landice.tests.dome.Dome The test group that this test case belongs to mesh_type : str The resolution or tye of mesh of the test case """ name = 'smoke_test' self.mesh_type = mesh_type subdir = '{}/{}'.format(mesh_type, name) super().__init__(test_group=test_group, name=name, subdir=subdir) self.add_step( SetupMesh(test_case=self, mesh_type=mesh_type)) self.add_step( RunModel(test_case=self, cores=4, threads=1, mesh_type=mesh_type)) step = Visualize(test_case=self, mesh_type=mesh_type) self.add_step(step, run_by_default=False) .. _imp_machine: Implementation: Machine-specific data ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis Machine-specific configuration options are in a set of config files under ``compass/machines``. As an example, the config file for Anvil looks like: .. code-block:: cfg # The paths section describes paths that are used within the ocean core test # cases. [paths] # The root to a location where the mesh_database, initial_condition_database, # and bathymetry_database for MPAS-Ocean will be cached ocean_database_root = /lcrc/group/e3sm/public_html/mpas_standalonedata/mpas-ocean # The root to a location where the mesh_database and initial_condition_database # for MALI will be cached landice_database_root = /lcrc/group/e3sm/public_html/mpas_standalonedata/mpas-albany-landice # the path to the base conda environment where compass environments have # been created compass_envs = /lcrc/soft/climate/e3sm-unified/base # The parallel section describes options related to running tests in parallel [parallel] # parallel system of execution: slurm or single_node system = slurm # whether to use mpirun or srun to run the model parallel_executable = srun # cores per node on the machine cores_per_node = 36 # the number of multiprocessing or dask threads to use threads = 18 It is likely that ``cores_per_node`` can be detected using a Slurm command and doesn't need to be supplied. This is something I have not fully explored yet. The ``threads`` option is not currently used and would also need to be explored. Additional config options are needed to support automatically generating job scripts, but this will be left for future work. The available machines are listed with: .. code-block:: bash python -m compass list --machine .. code-block:: none Machines: anvil badger default cori-haswell chrysalis cori-knl compy grizzly When setting up a test case or test suite, the ``--machine`` or ``-m`` flag is used to specify the machine. .. _imp_dir_struct: Implementation: Looser, more flexible directory structure ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis Test cases (and steps) in ``compass`` are uniquely defined by their relative paths within the work directory. The first two subdirectories in this path must be the name of the MPAS core and of the test group. The names and organization beyond that are quite flexible. Steps are expected to be nested somewhere within test cases but there is no restriction on the number of levels of subdirectories or their meaning beyond that of the test group. The idealized ocean test groups that have been implemented so far and the example test groups use the same organization as in legacy COMPASS: .. code-block:: none mpas_core/test_group/resolution/testcase/step For example: .. code-block:: none ocean/baroclinic_channel/10km/default/initial_state But the ``global_ocean`` test group takes advantage of the new flexibility. Here are the directories for test cases using the QU240 mesh: .. code-block:: none 27: ocean/global_ocean/QU240/mesh 28: ocean/global_ocean/QU240/PHC/init 29: ocean/global_ocean/QU240/PHC/performance_test 30: ocean/global_ocean/QU240/PHC/restart_test 31: ocean/global_ocean/QU240/PHC/decomp_test 32: ocean/global_ocean/QU240/PHC/threads_test 33: ocean/global_ocean/QU240/PHC/analysis_test 34: ocean/global_ocean/QU240/PHC/daily_output_test 35: ocean/global_ocean/QU240/PHC/dynamic_adjustment 36: ocean/global_ocean/QU240/PHC/files_for_e3sm 37: ocean/global_ocean/QU240/PHC/RK4/performance_test 38: ocean/global_ocean/QU240/PHC/RK4/restart_test 39: ocean/global_ocean/QU240/PHC/RK4/decomp_test 40: ocean/global_ocean/QU240/PHC/RK4/threads_test 41: ocean/global_ocean/QU240/EN4_1900/init 42: ocean/global_ocean/QU240/EN4_1900/performance_test 43: ocean/global_ocean/QU240/EN4_1900/dynamic_adjustment 44: ocean/global_ocean/QU240/EN4_1900/files_for_e3sm 45: ocean/global_ocean/QU240/PHC_BGC/init 46: ocean/global_ocean/QU240/PHC_BGC/performance_test As in legacy COMPASS, there is a subdirectory for the mesh. In the proposed design, there is a ``mesh`` test case with a single ``mesh`` step within that subdirectory. The mesh constructed and culled within that test case serves as the starting point for all other test cases using the mesh. Then, there are 4 different subdirectories for variants of the initial condition: either PHC or EN4_1900, and either with or without BGC. Each of these subdirectories has an ``init`` test case that creates the initial condition. The results of this test case are then used in all other steps within the subdirectory for that initial condition. Each remaining test case includes one or more forward model runs, or uses the results of such a run. Since the forward model can be run with either the split-explicit or the RK4 time integrator, variants of many test cases are supported with each time integrator. It is important that these are conceptually separate test cases because we use both the split-explicit and the RK4 versions of many test cases in our test suites. Each requires a set of corresponding namelist options and modifications to streams, so it is also not trivial for a user to switch between the two time integrators simply by manually modifying the test case at runtime. We treat the split-explicit time integrator as the default and put tests with RK4 in an additional ``RK4`` subdirectory. .. _imp_docs: Implementation: User- and developer-friendly documentation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis The documentation is still very much a work in progress and may be added with a separate pull request so that commits related to the infrastructure don't get intermixed with those for documentation. Documentation will continue to be generated automatically with Azure Pipelines using sphinx, as is the case for this design doc. The legacy COMPASS documentation will be renamed with "legacy" added to its titles (e.g. "Legacy User's Guide") and will be included at the end of the table of contents. The latest version of the test documentation is available in the branch: https://github.com/xylar/compass/tree/compass_1.0_docs and for browsing at the URL: https://mpas-dev.github.io/compass/test/index.html .. _imp_parallel: Implementation: Considerations related to running test cases in parallel ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis While an implementation of test-case parallelism will be left to a future design document and implementation, several parts of the current ``compass`` design and implementation were made with this work in mind. cores, max_memory and max_disk ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``Step`` base class keeps track of not only the number of cores (and threads) used but also the maximum allowed memory and disk. While the latter two are not currently used for anything and the values are just placeholders, they are expected to be useful for Parsl ``WorkerQueues``. inputs and outputs ~~~~~~~~~~~~~~~~~~ An effort has been made to be thorough about providing an absolute path to the inputs and outputs of each step. These are currently verified to make sure inputs are present before running a step and outputs are present after. We expect them to also be useful when we use Parsl to determine dependencies between steps and to figure out which can run in parallel with one another. .. _imp_res: Implementation: Resolution can be a test case parameter ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis This was discussed in :ref:`imp_dir_struct`. For all of the ocean and many of the land-ice test groups, either the resolution or the name of the mesh (which implicitly includes the resolution) is an argument to test case's and step's constructors. Nearly all test cases use that resolution or mesh name as a subdirectory within the relative path of the test case. So far, no convergence tests have been added where resolution is a parameter that varies across steps in a test case but the ``rpe_test`` test case of the ``baroclinic_channel`` includes viscosity as a parameter that varies across steps, and resolution is expected to be easy to use in the same way for future test cases. .. _imp_alter_code: Implementation: Test case code is easy to alter and rerun ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis When test cases and suites are set up from a local repository (and not the conda package from a conda environment), local symlinks to the ``compass`` directory are created. These links seem to provide and easy method for altering code and having it affect test cases and steps immediately without the need to build a conda package or a conda environment, or even to rerun ``python -m compass setup`` in most cases. .. _imp_premade_ic: Implementation: Support for pre-made initial condition files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/01/16 Contributors: Xylar Asay-Davis, Mark Petersen This work has not been included in any of the test cases that are part of the current implementation. Nothing in the implementation should preclude adding this capability later on. .. _imp_batch: Implementation: Easy batch submission ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/01/16 Contributors: Xylar Asay-Davis, Mark Petersen Batch scripts are not yet generated automatically as part of setting up a test case. Additional machine-specific config options will be needed to make this possible. This capability will be part of a future design. Nothing in the current implementation should preclude adding this capability later on. Indeed, it likely wouldn't be to much work. Testing ------- .. _test_easy: Testing: Make test cases easy to understand, modify and create ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis, Luke Van Roekel Given limited time, the reviewers will not attempt to implement any new MPAS cores or test groups as part of there reviews. However, in the near future, Luke Van Roekel has agreed to attempt to implement a test group ("single-column") and its test cases and steps as a test of the ease of understanding, modifying and creating test cases. Mark Petersen will add a new shallow-water MPAS core. Matt Hoffman will add new test cases as he has time and interest down the road. .. _test_shared_code: Testing: Shared code ^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis All of the test cases in the proposed implementation use shared code. Nearly all of the 86 test cases have been tested including those in all ocean and land-ice suites except the EC30to60 and ECwISC30to60. The 10 km RPE test for the ``baroclinic_channel`` test group has also been run successfully. The higher resolution versions of that test case have not yet been tested. So far, there is no indication of problems with shared code, but this is something of a subjective thing to test, beyond the proof of concept that code can, indeed, be shared. .. _test_shared_options: Testing: Shared configuration options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/01/16 Contributors: Xylar Asay-Davis All test cases include a config file and most test cases make use of config options from that file. I verified that altering the following config options in the ``ocean/global_ocean/QU240/PHC/init`` test case: .. code-block:: cfg [global_ocean] init_cores = 2 [vertical_grid] vert_levels = 32 max_layer_thickness = 250.0 Did indeed use 2 MPI tasks to produce an initial condition with 32 vertical levels, and a target maximum layer thickness of 250.0 m (actual was 245.35 m). It would be nearly impossible to test altering all parameters to see if they have the intended effect, so this will not be part of this testing. .. _test_core_count: Testing: Ability specify/modify core counts ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/01/16 Contributors: Xylar Asay-Davis This was included in :ref:`test_shared_options`. .. _test_machine_data: Testing: Machine-specific data ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/01/14 Contributors: Xylar Asay-Davis I ran the ocean nightly test suite on Anvil, providing ``-m anvil`` and no user config file. This worked successfully and no cached files were downloaded, meaning the cache directories were found successfully via Anvil's config file. I verified that the number of available cores and nodes in my job were successfully detected via Slurm commands. .. _test_dir_struct: Testing: Looser, more flexible directory structure ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis Testing of the ocean nightly suite includes tests of the flexible directory structure because it uses the ``global_ocean`` test group. More to the point, this capability has been tested by showing that test cases can be implemented using the flexible directory structure. .. _test_docs: Testing: User- and developer-friendly documentation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/04/13 Contributors: Xylar Asay-Davis Reviewers have been asked to run test cases and suites with the documentation, which is still evolving. Users and developers will be asked to run test cases and suites with the documentation and to add new test cases. In the near future, the documentation will be declared "good enough for now" and will be merged with the intention to update it on an ongoing basis. .. _test_parallel: Testing: Considerations related to running test cases in parallel ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/01/16 Contributors: Xylar Asay-Davis, Matt Hoffman This has not yet been implemented so will be tested as part of a later design. .. _test_res: Testing: Resolution can be a test case parameter ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/01/16 Contributors: Xylar Asay-Davis, Mark Petersen Resolution is a parameter in many existing test cases. No test case has yet been implemented that includes multiple steps with different resolutions so no testing of such a test case is possible at this time. .. _test_alter_code: Testing: Test case code is easy to alter and rerun ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/01/16 Contributors: Xylar Asay-Davis, Mark Petersen Xylar and Mark have both demonstrated that it is easy to modify code and rerun test cases without additional work because of the symlinks to the ``compass`` directory. .. _test_premade_ic: Testing: Support for pre-made initial condition files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/01/16 Contributors: Xylar Asay-Davis, Mark Petersen This was not yet implemented so no testing was performed. .. _test_batch: Testing: Easy batch submission ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Date last modified: 2021/01/16 Contributors: Xylar Asay-Davis, Mark Petersen This was not yet implemented so no testing was performed.