LANL IC overview and search

DST Calendar (within LANL network)

Information about Slurm:

Machine specifications: chicoma turquoise network

login: ssh -t <username> ssh ch-fe

File locations:

  • small home directory, for start-up scripts only: /users/<username>

  • home directory, backed up: /usr/projects/climate/<username>

  • scratch space, not backed up: /lustre/scratch4/turquoise/<username> or scratch5

Check compute time:

  • sacctmgr list assoc user=<username> format=Cluster,Account%18,Partition,QOS%45

  • Which is my default account? sacctmgr list user <username>

  • sshare -a | head -2; sshare -a | grep $ACCOUNT | head -1

  • sreport -t Hours cluster AccountUtilizationByUser start=2019-12-02 | grep $ACCOUNT

  • check job priority: sshare -a | head -2; sshare -a | grep $ACCOUNT

  • LANL Cluster Usage Overview (within LANL yellow)

Check disk usage:

  • your home space: chkhome

  • total disk usage in Petabytes: df -BP |head -n 1; df -BP|grep climate; df -BP |grep scratch


  • turquoise HPSS archive

  • archive front end: ssh -t <username> ssh ar-tn

  • storage available at: cd /archive/<project_name>

  • you can just copy files directly into here for a particular project.

LANL uses slurm. To obtain an interactive node:

salloc -N 1 -t 2:0:0 --qos=interactive

Use --account=ACCOUNT_NAME to change to a particular account.


There has not yet been a release with Chicoma-CPU, so the following applies to the release of compass v1.2.0, when it happens.

Chicoma’s CPU and GPU nodes have different configuration options and compilers. We only support Chicoma-CPU at this time.

config options

Here are the default config options added when you choose -m chicoma-cpu when setting up test cases or a test suite:

# The paths section describes paths that are used within the ocean core test
# cases.

# A shared root directory where MPAS standalone data can be found
database_root = /usr/projects/regionalclimate/COMMON_MPAS/mpas_standalonedata/

# the path to the base conda environment where compass environments have
# been created
compass_envs = /usr/projects/climate/SHARED_CLIMATE/compass/chicoma-cpu/base

# Options related to deploying a compass conda environment on supported
# machines

# the compiler set to use for system libraries and MPAS builds
compiler = gnu

# the system MPI library to use for gnu compiler
mpi_gnu = mpich

# the base path for spack environments used by compass
spack = /usr/projects/climate/SHARED_CLIMATE/compass/chicoma-cpu/spack

# whether to use the same modules for hdf5, netcdf-c, netcdf-fortran and
# pnetcdf as E3SM (spack modules are used otherwise)
use_e3sm_hdf5_netcdf = True

# The parallel section describes options related to running jobs in parallel

# account for running diagnostics jobs
account =

# cores per node on the machine
cores_per_node = 128

# threads per core (set to 1 because trying to hyperthread seems to be causing
# hanging on chicoma)
threads_per_core = 1

# Config options related to creating a job script

# The job partition to use
partition = standard

# The job quality of service (QOS) to use
qos = standard

Additionally, some relevant config options come from the mache package:

# The parallel section describes options related to running jobs in parallel

# parallel system of execution: slurm, cobalt or single_node
system = slurm

# whether to use mpirun or srun to run a task
parallel_executable = srun

# cores per node on the machine
cores_per_node = 256

# available partition(s) (default is the first)
partitions = standard, gpu

# quality of service (default is the first)
qos = standard, debug

# Config options related to spack environments

# whether to load modules from the spack yaml file before loading the spack
# environment
modules_before = False

# whether to load modules from the spack yaml file after loading the spack
# environment
modules_after = False

Gnu on Chicoma-CPU

To load the compass environment and modules, and set appropriate environment variables:

source /usr/projects/climate/SHARED_CLIMATE/compass/chicoma-cpu/

To build the MPAS model with

make [DEBUG=true] [OPENMP=true] [ALBANY=true] gnu-cray