Framework

All of the Packages and Modules that are not in the two cores (landice and ocean) belong to the compass framework. Some of these modules and packages are used by the Command-line interface, while others are meant to be called within test cases and steps to simplify tasks like adding input and output files, downloading data sets, building up config files, namelists and streams files, setting up and running the MPAS model, and verifying the output by comparing steps with one another or against a baseline.

list module

The compass.list.list_cases(), compass.list.list_machines() and compass.list.list_suites() functions are used by the compass list command to list test cases, supported machines and test suites, respectively. These functions are not currently used anywhere else in compass.

setup module

The compass.setup.setup_cases() and compass.setup.setup_case() functions are used by compass setup and compass suite to set up a list of test cases and a single test case, respectively, in a work directory. Subdirectories will be created for each test case and its steps; input, namelist and streams files will be downloaded, symlinked and/or generated in the setup process. A pickle file called test_case.pickle will be written to each test case directory containing the test-case object for later use in calls to compass run. Similarly, a file step.pickle containing both the step and test-case objects will be written to each step directory, allowing the step to be run on its own with compass run. In contrast to Config Files, these pickle files are not intended for users (or developers) to read or modify. Properties of the test-case and step objects are not intended to change between setting up and running a test suite, test case or step.

clean module

The compass.clean.clean_cases() function is used by compass clean and compass suite to delete the constants of a test-case subdirectory in the work directory.

suite module

The compass.suite.setup_suite() and compass.suite.clean_suite() functions are used by compass suite to set up or clean up a test suite in a work directory. Setting up a test suite includes setting up the test cases (see setup module), writing out a Provenance file, and saving a pickle file containing a python dictionary that defines the test suite for later use by compass run. The “target” and “minimum” number of cores required for running the test suite are displayed. The “target” is the maximum of the cores attribute of all steps in the test suite. This is the number of cores to run on to complete the test suite as quickly as possible, with the caveat that many cores may sit idle for some fraction of the runtime. The “minimum” number of cores is the maximum of the min_cores attribute for all steps int he suite, indicating the fewest cores that the test may be run with before at least some steps in the suite will fail.

run module

The compass.run.run_suite(), compass.run.run_test_case(), and compass.run.run_step() functions are used to run a test suite, test case or step, respectively, from the base, test case or step work directory, respectively, using compass run. Each of these functions reads a local pickle file to retrieve information about the test suite, test case and/or step that was stored during setup.

compass.run.run_suite() runs each test case in the test suite in the order that they are given in the text file defining the suite (compass/<mpas_core>/suites/<suite_name>.txt). It displays a PASS or FAIL message for the test execution, as well as similar messages for validation involving output within the test case or suite and validation against a baseline (depending on the implementation of the validate() method in the test case and whether a baseline was provided during setup). Output from test cases and their steps are stored in log files in the case_output subdirectory of the base work directory.

compass.run.run_test_case() and compass.run.run_step() run a single test case. In the latter case, only the selected step from the test case is run, skipping any others. If running the full test case, output from individual steps are stored in log files <step>.log in the test case’s work directory. The results of validation (if any) are displayed in the final stage of running the test case.

cache module

The compass.cache.update_cache() function is used by compass cache to copy step outputs to the compass_cache database on the LCRC server and to update <mpas_core>_cached_files.json files that contain a mapping between these cached files and the original outputs. This functionality enables running steps with Cached output files, which can be used to skip time-consuming initialization steps for faster development and debugging.

Config files

The compass.config module includes functions for creating and manipulating config options and Config Files.

The compass.config.add_config() function can be used to add the contents of a config file within a package to the current config parser. Examples of this can be found in most test cases as well as compass.setup.setup_case(). Here is a typical example from compass.landice.tests.enthalpy_benchmark.A.A.configure():

def configure(self):
    """
    Modify the configuration options for this test case
    """
    add_config(self.config, 'compass.landice.tests.enthalpy_benchmark.A',
               'A.cfg', exception=True)
    ...

The second and third arguments are the name of a package containing the config file and the name of the config file itself, respectively. You can see that the file is in the path compass/landice/tests/enthalpy_benchmark/A (replacing the . in the module name with /). In this case, we know that the config file should always exist, so we would like the code to raise an exception (exception=True) if the file is not found. This is the default behavior. In some cases, you would like the code to add the config options if the config file exists and do nothing if it does not. This can be useful if a common configure function is being used for all test cases in a configuration, as in this example from compass.ocean.tests.global_ocean.configure.configure_global_ocean():

add_config(config, test_case.__module__, '{}.cfg'.format(test_case.name),
           exception=False)

When this is called within the mesh test case, nothing will happen because compass/ocean/tests/global_ocean/mesh does not contain a mesh.cfg file. The config files for meshes are handled differently, since they aren’t associated with a particular test case:

mesh_step = mesh.mesh_step
add_config(config, mesh_step.package, mesh_step.mesh_config_filename,
           exception=True)

In this case, the mesh step keeps track of the package and config file in its attributes (e.g. compass.ocean.tests.global_ocean.mesh.qu240 and qu240.cfg for the QU240 and QUwISC240 meshes). Since we require each mesh to have config options (to define the vertical grid and the metadata to be added to the mesh, at the very least), we use exception=True so an exception will be raised if no config file is found.

The config module also contains 3 functions that are intended for internal use by the framework itself. Test-case developers will typically not need to call these functions directly.

The compass.config.duplicate_config() function can be used to make a deep copy of a config object so changes can be made without affecting the original.

The compass.config.ensure_absolute_paths() function is used internally by the framework to check and update config options in the paths, namelists, streams, and executables sections of the config file to make sure they have absolute paths. The absolute paths are determined from the location where one of the tools from the compass Command-line interface was called.

The compass.config.get_source_file() function is used to get an absolute path for a file using one of the config options defined in the paths section. This function is used by the framework as part of downloading files (e.g. to a defined database), see IO.

Logging

Compass does not have its own module for logging, instead making use of mpas_tools.logging. This is because a common strategy for logging to either stdout/stderr or to a log file is needed between compass and mpas_tools. To get details on how this module works in general, see MPAS-Tools’ Logging as well as the APIs for mpas_tools.logging.LoggingContext and mpas_tools.logging.check_call().

For the most part, the compass framework handles logging for you, so test-case developers won’t have to create their own logger objects. They are arguments to the test case’s run() or step’s run(). If you run a step on its own, no log file is created and logging happens to stdout/stderr. If you run the full test case, each step gets logged to its own log file within the test case’s work directory. If you run a test suite, each test case and its steps get logged to a file in the case_output directory of the suite’s work directory.

Although the logger will capture print statements, anywhere with a run() function or the functions called inside that function, it is a good idea to call logger.info instead of print to be explicit about the expectation that the output may go to a log file.

Even more important, subprocesses that produce output should always be called with mpas_tools.logging.check_call(), passing in the logger that is an argument to the run() function. Otherwise, output will go to stdout/stderr even when the intention is to write all output to a log file. Whereas logging can capture stdout/stderr to make sure that the print statements actually go to log files when desired, there is no similar trick for automatically capturing the output from direct calls to subprocess functions. Here is a code snippet from compass.landice.tests.dome.setup_mesh.SetupMesh.run():

from mpas_tools.logging import check_call


def run(self):
    ...
    section = config['dome']
    ...
    levels = section.getfloat('levels')
    args = ['create_landice_grid_from_generic_MPAS_grid.py',
            '-i', 'mpas_grid.nc',
            '-o', 'landice_grid.nc',
            '-l', levels]

    check_call(args, logger)
    ...

This example calls the script create_landice_grid_from_generic_MPAS_grid.py from mpas_tools with several arguments, making use of the logger.

IO

A lot of I/O related tasks are handled internally in the step class compass.Step. Some of the lower level functions can be called directly if need be.

Download

You can download files more directly if you need to using compass.io.download(), though we recommend using compass.Step.add_input_file() whenever possible because it is more flexible and takes care of more of the details of symlinking the local file and adding it as an input to the step. No current test cases use download() directly, but an example might look like this:

from compass.io import symlink, download

def setup(self):

    step_dir = self.work_dir
    database_root = self.config.get('paths', 'ocean_database_root')
    download_path = os.path.join(database_root, 'bathymetry_database')

    remote_filename = \
        'BedMachineAntarctica_and_GEBCO_2019_0.05_degree.200128.nc'
    local_filename = 'topography.nc'

    download(
        file_name=remote_filename,
        url='https://web.lcrc.anl.gov/public/e3sm/mpas_standalonedata/'
            'mpas-ocean/bathymetry_database',
        config=config, dest_path=download_path)

    symlink(os.path.join(download_path, remote_filename),
            os.path.join(step_dir, 'topography.nc'))

In this example, the remote file BedMachineAntarctica_and_GEBCO_2019_0.05_degree.200128.nc gets downloaded into the bathymetry database (if it’s not already there). Then, we create a local symlink called topography.nc to the file in the bathymetry database.

Model

Running MPAS

Steps that run the MPAS model should call the compass.Step.add_model_as_input() method from their __init__() method.

To run MPAS, call compass.model.run_model(). By default, this function first updates the namelist options associated with the PIO library and partitions the mesh across MPI tasks, as we will discuss in a moment, before running the model. You can provide non-default names for the graph, namelist and streams files. The number of cores and threads is determined from the cores, min_cores and threads attributes of the step object, set in its constructor or setup() method (i.e. before calling run()) so that the compass framework can ensure that the required resources are available.

Partitioning the mesh

The function compass.model.partition() calls the graph partitioning executable (gpmetis by default) to divide up the MPAS mesh across cores. If you call compass.model.run_model() with partition_graph=True (the default), this function is called automatically.

In some circumstances, a step may need to partition the mesh separately from running the model. Typically, this applies to cases where the model is run multiple times with the same partition and we don’t want to waste time creating the same partition over and over. For such cases, you can call compass.model.partition() and then provide partition_graph=False to later calls to compass.model.run_model().

Updating PIO namelist options

You can use compass.model.update_namelist_pio() to automatically set the MPAS namelist options config_pio_num_iotasks and config_pio_stride such that there is 1 PIO task per node of the MPAS run. This is particularly useful for PIO v1, which we have found performs much better in this configuration than when there is 1 PIO task per core, the MPAS default. When running with PIO v2, we have found little performance difference between the MPAS default and the compass default of one task per node, so we feel this is a safe default.

By default, this function is called within compass.model.run_model(). If the same namelist file is used for multiple model runs, it may be useful to update the number of PIO tasks only once. In this case, use update_pio=False when calling run_model(), then call compass.model.update_namelist_pio() yourself.

If you wish to use the MPAS default behavior of 1 PIO task per core, or wish to set config_pio_num_iotasks and config_pio_stride yourself, simply use update_pio=False when calling run_model().

Making a graph file

Some compass test cases take advantage of the fact that the MPAS-Tools cell culler can produce a graph file as part of the process of culling cells from an MPAS mesh. In test cases that do not require cells to be culled, you can call compass.model.make_graph_file() to produce a graph file from an MPAS mesh file. Optionally, you can provide the name of an MPAS field on cells in the mesh file that gives different weight to different cells (weight_field) in the partitioning process.

Validation

Test cases should typically include validation of variables and/or timers. This validation is a critical part of running test suites and comparing them to baselines.

Validating variables

The function compass.validate.compare_variables() can be used to compare variables in a file with a given relative path (filename1) with the same variables in another file (filename2) and/or against a baseline.

As a simple example:

variables = ['temperature', 'salinity', 'layerThickness', 'normalVelocity']
compare_variables(variables, config, work_dir=testcase['work_dir'],
                  filename1='forward/output.nc')

In this case, comparison will only take place if a baseline run is provided when the test case is set up (see compass setup or compass suite), since the keyword argument filename2 was not provided. If a baseline is provided, the 4 prognostic variables are compared between the file forward/output.nc and the same file in the corresponding location within the baseline.

Here is a slightly more complex example:

variables = ['temperature', 'salinity', 'layerThickness', 'normalVelocity']
compare_variables(variables, config, work_dir=testcase['work_dir'],
                  filename1='4proc/output.nc',
                  filename2='8proc/output.nc')

In this case, we compare the 4 prognostic variables in 4proc/output.nc with the same in 8proc/output.nc to make sure they are identical. If a baseline directory was provided, these 4 variables in each file will also be compared with those in the corresponding files in the baseline.

By default, the comparison will only be performed if both the 4proc and 8proc steps have been run (otherwise, we cannot be sure the data we want will be available). If one of the steps was not run (if the user is running steps one at a time or has altered the steps_to_run config option to remove some steps), the function will skip validation, logging a message that validation was not performed because of the missing step(s). You can pass the keyword argument skip_if_step_not_run=False to force validation to run (and possibly to fail because the output is not available) even if the user did not run the step involved in the validation.

In any of these cases, if comparison fails, the failure is stored in the validation attribute of the test case, and a ValueError will be raised later by the framework, terminating execution of the test case.

If quiet=False, typical output will look like this:

Beginning variable comparisons for all time levels of field 'temperature'. Note any time levels reported are 0-based.
    Pass thresholds are:
       L1: 0.00000000000000e+00
       L2: 0.00000000000000e+00
       L_Infinity: 0.00000000000000e+00
0:  l1: 0.00000000000000e+00  l2: 0.00000000000000e+00  linf: 0.00000000000000e+00
1:  l1: 0.00000000000000e+00  l2: 0.00000000000000e+00  linf: 0.00000000000000e+00
2:  l1: 0.00000000000000e+00  l2: 0.00000000000000e+00  linf: 0.00000000000000e+00
 ** PASS Comparison of temperature between /home/xylar/data/mpas/test_nightly_latest/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc and
    /home/xylar/data/mpas/test_nightly_latest/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc
Beginning variable comparisons for all time levels of field 'salinity'. Note any time levels reported are 0-based.
    Pass thresholds are:
       L1: 0.00000000000000e+00
       L2: 0.00000000000000e+00
       L_Infinity: 0.00000000000000e+00
0:  l1: 0.00000000000000e+00  l2: 0.00000000000000e+00  linf: 0.00000000000000e+00
1:  l1: 0.00000000000000e+00  l2: 0.00000000000000e+00  linf: 0.00000000000000e+00
2:  l1: 0.00000000000000e+00  l2: 0.00000000000000e+00  linf: 0.00000000000000e+00
 ** PASS Comparison of salinity between /home/xylar/data/mpas/test_nightly_latest/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc and
    /home/xylar/data/mpas/test_nightly_latest/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc
Beginning variable comparisons for all time levels of field 'layerThickness'. Note any time levels reported are 0-based.
    Pass thresholds are:
       L1: 0.00000000000000e+00
       L2: 0.00000000000000e+00
       L_Infinity: 0.00000000000000e+00
0:  l1: 0.00000000000000e+00  l2: 0.00000000000000e+00  linf: 0.00000000000000e+00
1:  l1: 0.00000000000000e+00  l2: 0.00000000000000e+00  linf: 0.00000000000000e+00
2:  l1: 0.00000000000000e+00  l2: 0.00000000000000e+00  linf: 0.00000000000000e+00
 ** PASS Comparison of layerThickness between /home/xylar/data/mpas/test_nightly_latest/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc and
    /home/xylar/data/mpas/test_nightly_latest/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc
Beginning variable comparisons for all time levels of field 'normalVelocity'. Note any time levels reported are 0-based.
    Pass thresholds are:
       L1: 0.00000000000000e+00
       L2: 0.00000000000000e+00
       L_Infinity: 0.00000000000000e+00
0:  l1: 0.00000000000000e+00  l2: 0.00000000000000e+00  linf: 0.00000000000000e+00
1:  l1: 0.00000000000000e+00  l2: 0.00000000000000e+00  linf: 0.00000000000000e+00
2:  l1: 0.00000000000000e+00  l2: 0.00000000000000e+00  linf: 0.00000000000000e+00
 ** PASS Comparison of normalVelocity between /home/xylar/data/mpas/test_nightly_latest/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc and
    /home/xylar/data/mpas/test_nightly_latest/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

If quiet=True (the default), there is only an indication that the comparison passed for each variable:

temperature          Time index: 0, 1, 2
  PASS /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc

       /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

salinity             Time index: 0, 1, 2
  PASS /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc

       /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

layerThickness       Time index: 0, 1, 2
  PASS /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc

       /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

normalVelocity       Time index: 0, 1, 2
  PASS /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc

       /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

temperature          Time index: 0, 1, 2
  PASS /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc

       /home/xylar/data/mpas/test_20210616/baseline/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc

salinity             Time index: 0, 1, 2
  PASS /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc

       /home/xylar/data/mpas/test_20210616/baseline/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc

layerThickness       Time index: 0, 1, 2
  PASS /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc

       /home/xylar/data/mpas/test_20210616/baseline/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc

normalVelocity       Time index: 0, 1, 2
  PASS /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc

       /home/xylar/data/mpas/test_20210616/baseline/ocean/baroclinic_channel/10km/threads_test/1thread/output.nc

temperature          Time index: 0, 1, 2
  PASS /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

       /home/xylar/data/mpas/test_20210616/baseline/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

salinity             Time index: 0, 1, 2
  PASS /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

       /home/xylar/data/mpas/test_20210616/baseline/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

layerThickness       Time index: 0, 1, 2
  PASS /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

       /home/xylar/data/mpas/test_20210616/baseline/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

normalVelocity       Time index: 0, 1, 2
  PASS /home/xylar/data/mpas/test_20210616/further_validation/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

       /home/xylar/data/mpas/test_20210616/baseline/ocean/baroclinic_channel/10km/threads_test/2thread/output.nc

By default, the function checks to make sure filename1 and, if provided, filename2 are output from one of the steps in the test case. In general, validation should be performed on outputs of the steps in this test case that are explicitly added with compass.Step.add_output_file(). This check can be disabled by setting check_outputs=False.

Norms

In the unlikely circumstance that you would like to allow comparison to pass with non-zero differences between variables, you can supply keyword arguments l1_norm, l2_norm and/or linf_norm to give the desired maximum values for these norms, above which the comparison will fail, raising a ValueError. These norms only affect the comparison between filename1 and filename2, not with the baseline (which always uses 0.0 for these norms). If you do want certain norms checked, you can pass their value as None.

If you want different nonzero norm values for different variables, the easiest solution is to call compass.validate.compare_variables() separately for each variable and with different norm values specified. compass.validate.compare_variables() can safely be called multiple times without clobbering a previous result. When you specify a nonzero norm, you may want compass to print the norm values it is using for comparison when the results are printed. To do so, use the optional quiet=False argument.

Validating timers

Timer validation is qualitatively similar to variable validation except that no errors are raised, meaning that the user must manually look at the comparison and make a judgment call about whether any changes in timing are large enough to indicate performance problems.

Calls to compass.validate.compare_timers() include a list of MPAS timers to compare and at least 1 directory where MPAS has been run and timers for the run are available.

Here is a typical call:

timers = ['time integration']
compare_timers(timers, config, work_dir, rundir1='forward')

Typical output will look like:

Comparing timer time integration:
             Base: 0.92264
          Compare: 0.82317
   Percent Change: -10.781019682649793%
          Speedup: 1.1208377370409515

Provenance

The compass.provenance module defines a function compass.provenance.write() for creating a file in the base work directory with provenance, such as the git version, conda packages, compass commands, and test cases.