Versions Compared

Old Version 31

changes.mady.by.user Louisa Emmons

Saved on

compared with

New Version Current

changes.mady.by.user Rebecca Buchholz

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

*** PAGE actively in development ***

...

Section


Column
width40%

You will need to ssh into 

Code Block
languagetext
derecho.hpc.ucar.edu



Column
width60%


Tip
titleTips from CISL:

Tips for moving from Cheyenne to Derecho on https://ncar-hpc-docs.readthedocs.io/


Note
titleComprehensive slides of information from NCAR/CISL about the Derecho HPC:

https://docs.google.com/presentation/d/1BAYI9NdyAnZzU3KLat0CK2sQj34XE_K5Wan9zL8rsQ0/edit?usp=sharing

Old instructions for running on Cheyenne (the retired NCAR HPC) are here: Run CAM-Chem on Cheyenne




CESM2.2 on Derecho

Your $HOME and $WORK directories (on glade) are the same path on cheyenne and derecho. Therefore, it is recommended to create a new directory (in $HOME or $WORK) for cases you will run on derecho, to not confuse them with cases built on cheyenne (which will not run on derecho).  Derecho and cheyenne have separate $SCRATCH directories.

...

(A)

...

Download a copy of the model

...

source code to your own directory

1. The first time, in your $HOME or $WORK directory, setup a folder to hold all your derecho cases.

...

Code Block
languagetext
> cd /glade/work/$USER/derecho_cases
> git clone https://github.com/ESCOMP/CESM.git cesm2_.2.2_derecho
> cd cesm2_.2.2_derecho
> git checkout cesm2.2.2-asdbranch

You should see in the $CESM_ROOT directory:

...

Get the optimized history-writing code for CLM with these commands: (Not sure this is still neeeded???)

Code Block
languagetext
> cd components/clm
> git fetch origin
> git checkout release-cesm2.2.01/hist_opt


If you want to run a regional refined model simulation on derecho, you may use the following sandbox in order to provide the same performance on derecho as it did on cheyenne:

Code Block
languagetext
/glade/work/fvitt/cesm/cesm2.2.2_musica

(B) Create a new case

2. Use the code in the model directory $CESM_ROOT to create a new case called $CASENAME:

...

Section


Column
width80%

7. (Optional) Check namelist settings in the namelist files user_nl_cam and user_nl_clm. Most CAM-chem related namelist variables are in CaseDocs/atm_in, but MEGAN and drydep are in CaseDocs/drv_flds_in (these files are created during build).  To modify any of these, copy the appropriate lines to user_nl_cam and edit there.

For example, if the startdate has been changed in env_run.xml, you have to also change the date of the initial meteorology file in user_nl_cam to start at the corresponding date.

For other changes see namelist changes or advanced options page.


Column
width20%


Tip
titleTip for New Users:

There are many namelist variables. You can find their definitions at:http://www.cesm.ucar.edu/models/cesm2/settings/current/cam_nml.html



After adding changes to user_nl_* files, optionally run:

Code Block
languagetext
>./preview_namelists

NOTE: most changes in user_nl_* files do not require re-building. However, during a run (CONTINUE_RUN = TRUE) no changes can be made to history output (fincl lists). If you want to change history output, create a new or branch run.

columnwidth
Note
titleNote

The pe-layout of existing compsets in the new code base of CESM2 for derecho (cesm2.2.2) has not been adjusted to the new computer, and running out of the box can lead to large differences in computer costs compared to running on cheyenne. 

New PE-layouts are still being developed. We are working on updating these compsets. One way to increase performance is to use a namelist setting. To improve improve the performance of derecho, you can add the following to your user_nl_cam file.

There are many namelist variables. You can find their definitions at:http://www.cesm.ucar.edu/models/cesm2/settings/current/cam_nml.html
 Code Block
languagetext
phys_loadbalance = 1
20%
 Tip
titleTip for New Users:


8. Check the run setup. In the env_batch.xml file make sure to have your project added correctly:  <entry id="PROJECT" value=$YOUR_PROJECT_CODE> Depending on the version of CESM, you may instead find the entry id for PROJECT in  env_workflow.xml
...
While running, output is written to <run_dir>: /glade/derecho/scratch/<username>/$CASENAME/run
 Section

 Column
width65%
After the run completes successfully, output files are moved to the short term archive: /glade/derecho/scratch/<username>/archive/$CASENAME/atm/hist (similar directories exist for other model components: lnd, etc.). 
Restart and initial conditions files are written to: /glade/derecho/scratch/<username>/archive/$CASENAME/rest
Note: long term archiving is currently not working
 Tip
titleTip for New Users:
For more options regarding the submission and running options type: case.submit --help


 Column
width35%

 Tip
titleTip for New Users:
Default output is monthly, so if you run a test 5-day simulation with monthly output, you will not see any files in the atm/hist location. However, restart files will have been created.


...