Troubleshooting

This section describes some common problems developers have run into and some suggested solutions.

Solver errors when deploying

When setting up compass pixi and Spack environments, compilers and system modules by calling:

./deploy.py ...

you may run into an error from pixi saying that it could not solve for the requested combination of packages. Details of the error vary, but the message usually names the packages whose version requirements conflict. The full output from deployment is in deploy_tmp/logs/.

If the error occurs while updating an existing deployment, the solution is usually to recreate the environment rather than trying to update it:

./deploy.py --recreate ...

The --recreate flag will first delete the existing pixi environment before creating it again with the new set of packages required for developing with the requested compiler and MPI type.

If the error also occurs with --recreate, the dependencies in deploy/pixi.toml.j2 and the pins in deploy/pins.cfg are likely incompatible with one another, which is a problem with the branch rather than with your environment.

Proxy on LANL Macs

If you are on a LANL Mac using the VPN and you see errors like:

An HTTP error occurred when trying to retrieve this URL.

this likely indicates that you need to set some proxy-related variables.

From a LANL internal site:

Bash, and other shells do not honor the system wide proxy settings on the Mac. There are a couple of ways for a person to manage these settings.

First, would be to the following will work in one’s ~/.profile - that way every time a shell is spawned with your username, these settings will be enabled:

proxy_enable() {
    export ALL_PROXY=proxyout.lanl.gov:8080
    export all_proxy=proxyout.lanl.gov:8080
    export NO_PROXY=localhost,127.0.0.1,*.lanl.gov,lanl.gov
    export no_proxy=localhost,127.0.0.1,*.lanl.gov,lanl.gov
}

proxy_disable() {
    unset ALL_PROXY
    unset all_proxy
    unset NO_PROXY
    unset no_proxy
}

proxy_enable

If you follow this approach, you would then call proxy_disable anytime you want to turn off the proxy (e.g. if the VPN is not running).

gethostbyname failed

If you see errors like the following (particularly on LANL Macs):

Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(159)..............:
MPID_Init(164).....................: channel initialization failed
MPIDI_CH3_Init(95).................:
MPID_nem_init(314).................:
MPID_nem_tcp_init(173).............:
MPID_nem_tcp_get_business_card(395):
GetSockInterfaceAddr(369)..........: gethostbyname failed, pn2034311.lanl.gov (errno 0)

this likely indicates that MPI is having a problem finding the local host.

The solution is to set the following config option in the parallel section of your user config file:

# The parallel section describes options related to running tests in parallel
[parallel]

# whether to use mpirun or srun to run the model
parallel_executable = mpirun -host localhost

The example config files have been updated to include this flag.