Versions Compared

Old Version 17

changes.mady.by.user Louisa Emmons

Saved on

compared with

New Version 18

changes.mady.by.user Louisa Emmons

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The following steps describe creating a new grid and using it in a beta version of CESM3.  They make use of existing executables on derecho or casper.  More general instructions Instructions for CESM2.2 are available in the the step-by-step guide.

For instructions on connecting to derecho or casper see: https://ncar.github.io/CESM-Tutorial/notebooks/prereqs/prereqs_overview.html   

Naming Conventions

$(grid_name) = ne0np4.NAME.ne30xR where NAME is a unique name for your grid, ne30 is for the base resolution,  R is for the refinement factor of the highest resolution region.  All variable resolution grids with 4 GLL points must begin with ne0np4.

$(grid_label) = NAME_ne30xR

In the examples below, a grid for a small refined region at ne30x8 (ne240, 1/8 deg) centered over Nanjing will be created, using NAME=Nanjing, R=8, so:

  • $(grid_name) = ne0np4.Nanjing.ne30x8 ne30x8 
  • $(grid_label) = Nanjing_ne30x8

Replace these with appropriate names and resolution for your own grid.

Set up a repository for your grid

...

Create ESMF mesh file from SCRIP file

On casper, load needed modules:

> module load mpi-serial/2.3.0
> module load esmf/8.5.0

...

In the directory with your SCRIP file, run (be sure all on one line):

>

...

/glade/u/apps/casper/23.10/spack/opt/spack/esmf/8.5.0/mpi-serial/2.3.0/oneapi/2023.2.1/dfkx/bin/ESMF_Scrip2Unstruct

...

Nanjing_ne30x8_np4_SCRIP.nc

...

Nanjing_ne30x8_np4_MESH.nc

...

0

Note the 0 at the end.

This creates the ESMF mesh file: Nanjing_ne30x8_np4_MESH.nc

...

To keep with previous conventions for the structure of your new grid repository create directories inic, maps, atmsrf and topo in your repository:

...

Edit the script to point to your grid files.  This template includes the path to an existing executable of interpic, which is available in the CESM source code.

# USER CHANGES
VRdate="YYMMDD"
VRgridName="ne0np4.NAME.ne30xR"
VRgridLabel="NAME_RESOL"
VRrepoPath="your_repo_path"
# end of USER CHANGES

...

The resulting file is assigned to bnd_topo in user_nl_cam.

> qcmd -l walltime=12:00:00 -l select=1:ncpus=1 -- /glade/work/emmons/Topo_derecho/cube_to_target/cube_to_target --rrfac_manipulation --grid_descriptor_file='/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_np4_SCRIP.nc' --intermediate_cs_name='/glade/campaign/cgd/amp/pel/topo/cubedata/gmted2010_modis_bedmachine-ncube3000-220518.nc' --output_grid='Nanjing_ne30x8' --rrfac_max=8 --smoothing_scale=100.0 -u 'Louisa Emmons, emmons@ucar.edu' > out_Nanjing

This can take several hours.  (to do: set up to run as a submitted job with qsub)

Set up CESM3 with new grid

Set up a CAM case for testing new grid

...

The following steps get the Topo software (https://github.com/NCAR/Topo), build the cube_to_target executable and run it. Documentation for Topo is available on its wiki page.

On casper:

> cd $REPO
> git clone https://github.com/NCAR/Topo.git Topo
> cd Topo
> cd cube_to_target
> module load gcc
> module list

Currently Loaded Modules:

  1) ncarenv/23.10 (S)   2) ncl/6.6.2   3) gcc/12.2.0   4) ncarcompilers/1.0.0   5) hdf5/1.12.2   6) netcdf/4.9.2   7) cuda/12.2.1   8) ucx/1.14.1   9) openmpi/4.1.6  10) esmf/8.5.0

> gmake -f Makefile clean
> gmake -f Makefile

Put following in a script or be sure entire command is on one line:

> qcmd -l walltime=12:00:00 -l select=1:ncpus=1 -- ./cube_to_target --rrfac_manipulation
--grid_descriptor_file='/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_np4_SCRIP.nc'
--intermediate_cs_name='/glade/campaign/cgd/amp/pel/topo/cubedata/gmted2010_modis_bedmachine-ncube3000-220518.nc'
--output_grid='Nanjing_ne30x8' --rrfac_max=8 --smoothing_scale=100.0 -u 'Louisa Emmons, emmons@ucar.edu' > out_Nanjing

Modify the bold text to point to your SCRIP file, grid name, etc.

This can take several hours.  (to do: set up to run as a submitted job with qsub)

Generate CTSM (CLM) surface datasets

On derecho, use CTSM5.2 source code to create the needed surfdata and landuse_timeseries files.

> git clone https://github.com/ESCOMP/CTSM ctsm5.2.007
> cd ctsm5.2.007
> git checkout ctsm5.2.007
> bin/git-fleximod update

Build executable for mksurfdata

> cd ./tools/mksurfdata_esmf
> ./gen_mksurfdata_build

Generate namelist files containing specifications for creating the surface datasets.  

First load the conda environment:

> module load conda/latest
> conda activate npl

Run the make namelist script (modify the bold text for your MESH file and your ncol for model-mesh-nx)

(npl) > /glade/work/emmons/cesm_src_derecho/ctsm5.2.007/tools/mksurfdata_esmf/gen_mksurfdata_namelist --res Nanjing_ne30x8 --start-year 1979 --end-year 2026 --ssp-rcp SSP3-7.0 --model-mesh /glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_np4_MESH.nc --model-mesh-nx 60482 --model-mesh-ny 1

This creates files like: landuse_timeseries_SSP3-7.0_1979-2026_78pfts.txt and surfdata_Nanjing_ne30x8_SSP3-7.0_1979_78pfts_c240809.namelist.

Now create a job script (use your namelist filename):

> /glade/work/emmons/cesm_src_derecho/ctsm5.2.007/tools/mksurfdata_esmf/gen_mksurfdata_jobscript_single --number-of-nodes 4 --tasks-per-node 128 --namelist-file surfdata_Nanjing_ne30x8_SSP3-7.0_1979_78pfts_c240809.namelist

This creates mksurfdata_jobscript_single.sh.  Edit this file to use your project # in '#PBS -A P...'.

> qsub mksurfdata_jobscript_single.sh

This creates the netcdf files that will be used in your simulation, specified in user_nl_clm: fsurdat = '/repo_path/land/surfdata_ ... .nc' and flanduse_timeseries = '/repo_path/land/landuse.timeseries_ ... .nc'.  In your $REPO, create a directory 'land' (mkdir $REPO/land) and copy these 2 nc files to it.

> conda deactivate

Set up CESM3 with new grid

On derecho, checkout a CESM3 beta tag (that uses ctsm5.2).  It is recommended the source code is cloned into your work (or scratch, if necessary) directory where sufficient storage is available. CESM3 has replaced manage_externals with git-fleximod, which requires the following commands:

> git clone https://github.com/ESCOMP/CESM.git cesm3_0_beta01
> cd cesm3_0_beta01
> git checkout cesm3_0_beta01
> ./bin/git-fleximod update

Below, <cesm> refers to yourcesm3_0_beta01 directory.

CCS_CONFIG

  1. Add an entry for your new ESMF mesh file in <cesm>/ccs_config/component_grids_nuopc.xml

<domain name="(grid_name)">

    <nx>(ncol)</nx> <ny>1</ny>

    <mesh>/your_path/(new_grid)_ESMFmesh.nc</mesh>

    <desc>(new_grid) is a Spectral Elem 1-deg grid with a 1/NN deg refined region over …:</desc>

    <support>Test support only</support>

  </domain>

Include the corresponding number of grid columns in the <nx> argument, and a proper <domain_name>.


2. Add a grid alias entry in <cesm>/ccs_config/modelgrid_aliases_nuopc.xml:

  <model_grid alias="ne0ARCTICne30x4_ne0ARCTICne30x4_mt13" not_compset="_POP">
    <grid name="atm">ne0np4.ARCTIC.ne30x4</grid>
    <grid name="lnd">ne0np4.ARCTIC.ne30x4</grid>
    <grid name="ocnice">ne0np4.ARCTIC.ne30x4</grid>
    <mask>tx0.1v3</mask>
  </model_grid>

Note the “grid name” must match the ”domain name”  in component_grids_nuopc.xml.

Set up a CAM case for testing new grid

Create a new case with FHIST compset:

> <cesm>/cime/scripts/create_newcase ...

> cd <case_dir>

> ./xmlchange NTASKS= ___

> ./case.setup

Add to user_nl_cam:

ncdata=‘<path_to_ncdata>’

bnd_topo=‘<path_to_topo>’

drydep_srf_file=‘<path_to_drydep>’

se_refined_mesh=.true.

se_mesh_file=‘<path_to_exodus>’

inithist = 'DAILY'

Set the timestep based on recommended value in: https://github.com/ESMCI/Community_Mesh_Generation_Toolkit/blob/master/VRM_tools/Docs/CAM-tsteps-inic-for-newgrids_v0.pdf

./xmlchangeATM_NCPL=XXX


Add to user_nl_clm

fsurdat=‘<path_to_surfdata>’

flanduse_timeseries=‘<path_to_landuse_timeseries>’

Add the override flag in your individual case directory to ensure CTSM doesn’t error out due to an unsupported grid:

./xmlchange --append CLM_BLDNML_OPTS="-no-chk_res"


> qcmd -- ./case.build

Set up to run a few days. 

If it does not run, try adjusting parameters, as described in https://github.com/ESMCI/Community_Mesh_Generation_Toolkit/blob/master/VRM_tools/Docs/CAM-tsteps-inic-for-newgrids_v0.pdf 

Once it runs, change in user_nl_cam:

inithist = 'MONTHLY'

and  run a year.  Save the final *cam.i.* file to use for future CAM simulations (ncdata in user_nl_cam). 

Also, save the final CLM restart file (*.clm2.r.*.nc) to use for finidat in user_nl_clm in future runs.

Set up a CAM-chem case for production

Create a case with FCnudged compset.

You will need to regrid a previous CAM-chem IC file to your grid (to use for ncdata in user_nl_cam).

Also, regrid emissions and met files to your grid.

Use the CLM restart file from your CAM spinup for finidat in user_nl_clm.