The following steps describe creating a new grid and using it in a beta version of CESM3. They make use of existing executables on derecho or casper. More general instructions Instructions for CESM2.2 are available in the the step-by-step guide.

For instructions on connecting to derecho or casper see: https://ncar.github.io/CESM-Tutorial/notebooks/prereqs/prereqs_overview.html

Table of Contents

maxLevel	1

Anchor
Intro
Intro
Naming Conventions

$(grid_name) = ne0np4.NAME.ne30xR where NAME is a unique name for your grid, ne30 is for the base resolution, R is for the refinement factor of the highest resolution region. All variable resolution grids with 4 GLL points must begin with ne0np4.

$(grid_label) = NAME_ne30xR

In the examples below, a grid for a small refined region at ne30x8 (ne240, 1/8 deg) centered over Nanjing will be created, using NAME=Nanjing, R=8, so:

$(grid_name) = ne0np4.Nanjing.ne30x8 ne30x8
$(grid_label) = Nanjing_ne30x8

Replace these with appropriate names and resolution for your own grid.

Set up a repository for your grid

...

$REPO = /glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/

Anchor
Create Grid
Create Grid
Create a new variable resolution mesh (VRM)

In your repository directory, create a 'grids' subdirectory.

On casper start the VRM Editor from your $REPO/grids directory:

> cd $REPO/grids
> /glade/work/emmons/tutorial_Nanjing/VRM_tools/VRM_Editor/src/VRM_Editor

If you are working on a different computer, download the VRM_tools software by running:

...

Get and build VRM Tools: VRM_Editor and Create_VRMgrid

On casper, download the VRM Tools software:

{navigate to your home or work directory}
> git clone https://github.com/ESMCI/Community_Mesh_Generation_Toolkit.git
> cd

...

Community_Mesh_Generation_Toolkit

...

and follow instructions in the documentation.

Set up the base grid

On the ‘VRM’ tab, select:

Grid Type: CubeSquared
Base Resolution: 30
Click Generate VarMesh. See image below for what it should look like.

Image Removed

Adjust Longitude Shift, Rotate-X and Rotate-Y so that your region of interest is centered within a cube face. You can zoom in/out by using the scroll button of your mouse. The right and left arrow keys move the map sideways; up/down to shift the map N/S.

For example, set Longitude Shift to 30 and Rotate-X to 25, then click Generate VarMesh, to produce grid shown below.

Image Removed

When satisfied with the face location, record the Longitude Shift, Rotate-X and Rotate-Y values (you will need them later).

Create draft refinement region

The default resolution of the Editor map is a bit coarse, so read in a higher resolution map. In the "Actions" menu (upper left), select Read Refinement Map.

...


> git checkout VRMtools_MCT_v1.0

In the instructions below, $(VRM_tools) = {your_path}/Community_Mesh_Generation_Toolkit/VRM_tools

> cd ${VRM_tools}/VRM_Editor/src
> module load gcc/12.2.0 
> module load ncarenv/23.10
> qmake VRM_Editor.pro
> make

Running "make" produces a lot of warning messages; that’s ok. For Linux users, with the Qt and netCDF packages installed, building the editor should be as simple as for Casper.

Build Create_VRMgrid on Casper:

> cd Community_Mesh_Generation_Toolkit/VRM_tools/VRM_Editor/src
> module load gnu
> make -f Makefile-Create_VRMgrid

Instructions on using it are provided below.

Start VRM_Editor

In your repository directory, create a 'grids' subdirectory. On casper start the VRM Editor from your $REPO/grids directory:

> cd $REPO/grids
> ${VRM_tools}/VRM_Editor/src/

...

VRM_Editor

Set up base grid

On the ‘Edit’ tab, click ‘Edit refine Map’.‘VRM’ tab, select:

Grid Type: CubeSquared
Base Resolution: 30
Click Generate VarMesh. See image below for what it should look like.

Image Added

Adjust Longitude Shift, Rotate-X and Rotate-Y so that your region of interest is centered within a cube face. You can zoom in/out by using Select ‘Polygon Editor’ - a green box appears over the Pacific. Drag and adjust the 8 points to create a refinement region. Use the scroll button of your mouse to zoom in and out, and the arrow keys . The right and left arrow keys move the map sideways; up/down to shift the map . Try to have the polygon borders be parallel to the grid lines.

Image Removed

Click ‘Apply’ (with Value =1.00)

Image Removed

To save this, click 'Exit Edit Mode' and 'Yes' to save.

Go to 'Display' tab, check Refinement Map to see the refined region as a red box.

Go back to ‘VRM’ tab. For best results (in many cases), the following settings are recommended:

Refine Type = LOWCONN
Select Refine Level (1 = ~0.5 deg, 2=0.25 deg, 3=⅛ deg=14km, …). Resolution in the refined region is the base grid resolution divided by 2^(refinement level)
Smoothing Options: Spring, with Iterations = 3, Length = 3

Click 'Generate VarMesh'. This case has Refine Lev = 3, to give a refined mesh with 1/8 degree resolution at center:

Image Removed

The LOWCONN setting uses templates that span 2x2 base elements to transition between resolutions, and the SPRING smoothing rounds out the element shapes to reduce sharp angles. There may be situations when other settings give better results.

Uneven edges might be removed with small adjustments (0.1) to Longitude Shift or Rotate-X, -Y values.

It is best to have a couple of rows at each intermediate resolution. This can be done by making a 'halo' in the Editor: make the polygon a bit larger than the refined region, and set the value at the appropriate fraction. For an ne30 base grid with Refine Level=3 (ne240), the halo region surrounding the finest refinement (at ne120 resolution) should have a Value=0.66.

Image Removed

Then a second halo, at ne60 resolution, can be created with Value=0.33.

Image Removed

If making a refined region with the finest resolution ne120, then make one halo with Value=0.5.

When satisfied with the halos, Exit Edit Mode, Yes to save, and go back to VRM tab. With the same settings as above (LOWCONN, Refine lev =3, etc.), click Generate VarMesh:

Image Removed

The halos can be adjusted repeatedly in the Edit menu until satisfied with the grid. Once the grid is at least close to what you want proceed to steps below.

Save the Refinement Map - under the Actions menu. This writes a netcdf file of the map of refinement values (0 for no refinement, 1 for maximum refinement). Save the files as something like: REFMAP_Nanjing_ne30x8.nc.

Save frequently as the VRM Editor on the Mac tends to crash. You can then Read the refinement later and start adjusting the grid from that point.

Also, make note of any Longitude Shift and Rotate-X, -Y values as those are not saved in the Refinement map file.

If you are happy with your grid, under Actions: Write Exodus File - give it a name like Nanjing_ne30x8_EXODUS.nc. If you want to make further manual edits to your grid, do not save the EXODUS file.

Write Refinement Grid - for manual editing of refinement region and halos

If you have trouble making a grid with smooth borders and transitions around the refinement region, you can save the grid as a text file and manually edit it.

Once you have a rough version of your grid, save a Refinement Grid - in the Actions menu, select 'Write Refinement Grid'.

This is a text file similar to the Refinement Map, so name it something like: RefGrid_Nanjing_ne30x8.dat

Record the Longitude Shift, Rotate-X and Rotate-Y values.

Then edit the RefGrid file to have straight edges for each refine level. For a refinement factor of 8 (refine level=3) the file has 0 for no refinement and 1, 2, 3, etc. for each refine level.

This example shows irregular borders, and different width halos:

Image Removed

After editing, a clean grid template looks like this:

Image Removed

Use Create_VRMgrid to create EXODUS file from Refinement Grid

The command line ‘Create_VRMgrid’ will create an EXODUS file from the updated Refinement Grid file.

> /glade/work/emmons/tutorial_Nanjing/VRM_tools/VRM_Editor/src/Create_VRMgrid 
--refine_type "LOWCONN" {or "CUBIT"} 
--grid_type "CubeSquared" 
--resolution 30
--refine_level {refinement [1,2,3 …]}
--smooth_type "SPRING" --smooth_dist 3 --smooth_iter 3 
--x_rotate {xrot_value} --y_rotate {yrot_value} --lon_shift {lonshift_value} 
--refine_file REFMAP_{yourLabel}_{resolution}.nc
--refine_cube RefGrid_{yourLabel}_{resolution}.dat 
--output {yourLabel}_{resolution}_EXODUS.nc

(be sure this is all on one line)

For this example: /glade/derecho/scratch/emmons/nanjing_musica_tutorial> /glade/work/emmons/tutorial_Nanjing/VRM_tools/VRM_Editor/src/Create_VRMgrid --refine_type "LOWCONN" --grid_type "CubeSquared" --resolution 30 --refine_level 3 --smooth_type "SPRING" --smooth_dist 3 --smooth_iter 3 --x_rotate 25 --y_rotate 0 --lon_shift 30 --refine_file REFMAP_Nanjing_ne30x8.nc --refine_cube RefGrid_Nanjing_ne30x8.dat --output Nanjing_ne30x8_EXODUS.nc

Plot EXODUS file with /glade/u/home/emmons/tutorial_Nanjing_notebooks/Plot_exodus_grid.ipynb

Image Removed

Create grid files

Several types of grid files need to be created. These grids are described on the overview page.

Create SCRIP and LATLON grid files from EXODUS file

On casper:

> cd $REPO/grids

N/S.

For example, set Longitude Shift to 30 and Rotate-X to 25, then click Generate VarMesh, to produce grid shown below.

Image Added

When satisfied with the face location, record the Longitude Shift, Rotate-X and Rotate-Y values (you will need them later).

Create draft refinement region

The default resolution of the Editor map is a bit coarse, so read in a higher resolution map. In the "Actions" menu (upper left), select Read Refinement Map.

Open /glade/work/emmons/Community_Mesh_Generation_Toolkit_Nov23/VRM_tools/VRM_Editor/src/REFMAP_1440x720.nc. For easier access, copy this file to your working directory.

On the ‘Edit’ tab, click ‘Edit refine Map’.

Select ‘Polygon Editor’ - a green box appears over the Pacific. Drag and adjust the 8 points to create a refinement region. Use the scroll button of your mouse to zoom in and out, and the arrow keys to shift the map. Try to have the polygon borders be parallel to the grid lines.

Image Added

Click ‘Apply’ (with Value =1.00)

Image Added

To save this, click 'Exit Edit Mode' and 'Yes' to save.

Go to 'Display' tab, check Refinement Map to see the refined region as a red box.

Go back to ‘VRM’ tab. For best results (in many cases), the following settings are recommended:

Refine Type = LOWCONN
Select Refine Level (1 = ~0.5 deg, 2=0.25 deg, 3=⅛ deg=14km, …). Resolution in the refined region is the base grid resolution divided by 2^(refinement level)
Smoothing Options: Spring, with Iterations = 3, Length = 3

Click 'Generate VarMesh'. This case has Refine Lev = 3, to give a refined mesh with 1/8 degree resolution at center:

Image Added

The LOWCONN setting uses templates that span 2x2 base elements to transition between resolutions, and the SPRING smoothing rounds out the element shapes to reduce sharp angles. There may be situations when other settings give better results.

Uneven edges might be removed with small adjustments (0.1) to Longitude Shift or Rotate-X, -Y values.

It is best to have a couple of rows at each intermediate resolution. This can be done by making a 'halo' in the Editor: make the polygon a bit larger than the refined region, and set the value at the appropriate fraction. For an ne30 base grid with Refine Level=3 (ne240), the halo region surrounding the finest refinement (at ne120 resolution) should have a Value=0.66.

Image Added

Then a second halo, at ne60 resolution, can be created with Value=0.33.

Image Added

If making a refined region with a refine level =2 (resolution ne120), then make one halo with Value=0.5.

When satisfied with the halos, Exit Edit Mode, Yes to save, and go back to VRM tab. With the same settings as used before (LOWCONN, Refine level =3, etc.), click Generate VarMesh:

Image Added

The halos can be adjusted repeatedly in the Edit menu until satisfied with the grid. You can try "CUBIT" instead of "LOWCONN" to see if that is more appropriate for your application.

Once the grid is at least close to what you want proceed to steps below.

1) Save the Refinement Map - under the Actions menu. This writes a netcdf file of the map of refinement values (0 for no refinement, 1 for maximum refinement). Save the files as something like: REFMAP_Nanjing_ne30x8.nc.

Save frequently as the VRM Editor tends to crash. You can then restart the Editor, Read the Refinement Map and start adjusting the grid from that point.

Also, make note of any Longitude Shift and Rotate-X, -Y values as those are not saved in the Refinement Map file.

2) Write EXODUS File: If you are happy with your grid, under Actions: Write Exodus File - give it a name like Nanjing_ne30x8_EXODUS.nc.

If you want to make further manual edits to your grid, as described below, do not save the EXODUS file.

3) Write Refinement Grid - for manual editing of refinement region and halos

If you have trouble making a grid with smooth borders and transitions around the refinement region, you can save the grid as a text file and manually edit it.

Once you have a rough version of your grid, save the Refinement Grid - in the Actions menu, select 'Write Refinement Grid'.

This is a text file similar to the Refinement Map, so name it something like: RefGrid_Nanjing_ne30x8.dat

Record the Longitude Shift, Rotate-X and Rotate-Y values.

Then edit the RefGrid file to have straight edges for each refine level. For a refinement factor of 8 (refine level=3) the file has 0 for no refinement and 1, 2, 3, etc. for each refine level.

This example shows irregular borders, and different width halos:

Image Added

After editing, a clean grid template looks like this:

Image Added

Use Create_VRMgrid to create EXODUS file from Refinement Grid

The command line ‘Create_VRMgrid’ will create an EXODUS file from the updated Refinement Grid file.

> /glade/work/emmons/tutorial_Nanjing/VRM_tools/VRM_Editor/src/Create_VRMgrid 
--refine_type "LOWCONN" {or "CUBIT"} 
--grid_type "CubeSquared" 
--resolution 30
--refine_level {refinement [1,2,3 …]}
--smooth_type "SPRING" --smooth_dist 3 --smooth_iter 3 
--x_rotate {xrot_value} --y_rotate {yrot_value} --lon_shift {lonshift_value} 
--refine_file REFMAP_{yourLabel}_{resolution}.nc
--refine_cube RefGrid_{yourLabel}_{resolution}.dat 
--output {yourLabel}_{resolution}_EXODUS.nc

(be sure this is all on one line)

For this example:

> /glade/work/emmons/tutorial_Nanjing/VRM_tools/VRM_Editor/src/Create_VRMgrid --refine_type "LOWCONN" --grid_type "CubeSquared" --resolution 30 --refine_level 3 --smooth_type "SPRING" --smooth_dist 3 --smooth_iter 3 --x_rotate 25 --y_rotate 0 --lon_shift 30 --refine_file REFMAP_Nanjing_ne30x8.nc --refine_cube RefGrid_Nanjing_ne30x8.dat --output Nanjing_ne30x8_EXODUS.nc

Plot EXODUS file with /glade/u/home/emmons/tutorial_Nanjing_notebooks/plotting/Plot_exodus_grid.ipynb

Image Added

Anchor
Create mesh files
Create mesh files
Create grid files

Several types of grid files need to be created. These grids are described on the overview page.

Create SCRIP and LATLON grid files from EXODUS file

On casper: build Gen_ControlVolumes.exe; be sure to use the gnu (gcc) compiler.

> cd $(VRM_tools)/VRM_ControlVolumes/src
> module load gcc
> make
> module load intel

> cd $REPO/grids

> cp ${VRM_tools}/VRM_ControlVolumes/src/input.nl input-Nanjing.nl

Edit input-Nanjing.nl to have your path to $REPO/grids and the grid name.

Then run Gen_ControlVolumes:

> ${VRM_tools}/VRM_ControlVolumes/src/Gen_ControlVolumes.exe input-Nanjing.nl > LOG_Nanjing

This produces the SCRIP file: Nanjing_ne30x8_np4_SCRIP.nc which is used for further regridding steps, as well as for plotting your final model output on the native grid.

A "LATLON" file is also produced: Nanjing_ne30x8_np4_LATLON.nc

If you do not see these files, check LOG_Nanjing for errors.

Examine the LATLON file to get the number of columns in your new grid:

> ncdump -h Nanjing_ne30x8_np4_LATLON.nc

This prints (among other things):

ncol = 60482 ;

You will need this value for several of the following steps.

Create ESMF mesh file from SCRIP file

On casper, load needed modules:

> module load mpi-serial/2.3.0
> module load esmf/8.5.0

To find the path toESMF_Scrip2Unstruct, run:

> module show esmf/8.5.0

Find the listing for the "PATH" directory:

"PATH","/glade/u/apps/casper/23.10/spack/opt/spack/esmf/8.5.0/mpi-serial/2.3.0/oneapi/2023.2.1/dfkx/bin"

In the directory with your SCRIP file ($REPO/grids), run (be sure all on one line):

> /glade/u/apps/casper/23.10/spack/opt/spack/esmf/8.5.0/mpi-serial/2.3.0/oneapi/2023.2.1/dfkx/bin/ESMF_Scrip2Unstruct Nanjing_ne30x8_np4_SCRIP.nc Nanjing_ne30x8_np4_MESH.nc 0

Note the 0 at the end.

This creates the ESMF mesh file: Nanjing_ne30x8_np4_MESH.nc

Anchor
Input files
Input files
Generate CESM input files on new grid

To keep with previous conventions for the structure of your new grid repository create directories inic, maps, atmsrf and topo in your repository:

> cd $REPO
> mkdir inic
> mkdir maps
> mkdir atmsrf
> mkdir topo

The files created below are generally labeled with today's date. Even if you make files on different days, all the files in one repository should have the same date (YYMMDD).

Regrid CAM IC file

The resulting file is assigned to ncdata in user_nl_cam.

This script uses the interpic program provided in CESM source code, but must first be compiled. If you do not already have CESM code, see "Set up CESM3" below.

On casper, in CESM source code, ./components/cam/tools/interpic_new:

Edit Makefile:
l.15 from: LIB_NETCDF := /usr/local/lib
to: LIB_NETCDF := $(NETCDF)/lib
l.18 from: INC_NETCDF := /usr/local/include
to: INC_NETCDF := $(NETCDF)/include
l.99 from: LDFLAGS = -L$(LIB_NETCDF) -lnetcdf
to: LDFLAGS = -L$(LIB_NETCDF) -lnetcdff -lnetcdf

Currently Loaded Modules:
1) ncarenv/23.10 (S) 2) intel/2023.2.1 3) ncarcompilers/1.0.0 4) hdf5/1.12.2 5) netcdf/4.9.2

> gmake

This should have created the executable ‘interpic’. Alternatively, the instructions below use an existing executable available on casper.

> cd $REPO/inic
> cp /glade/work/emmons/tutorial_Nanjing/VRM_tools/gen_CAMncdata/TEMPLATES/interpic_script_TEMPLATE.sh interpic_script_Nanjing.sh
> vi interpic_script_Nanjing.sh

Edit the script to point to your grid files. This template includes the path to an existing executable of interpic, which is available in the CESM source code.

# USER CHANGES
VRdate="YYMMDD"
VRgridName="ne0np4.NAME.ne30xR"
VRgridLabel="NAME_RESOL"
VRrepoPath="your_repo_path"
# end of USER CHANGES

On casper:

> module load nco
> module load ncl
> qcmd -- 'sh interpic_script_Nanjing.sh > log_Nanjing'

Check there are no errors in the log file. This should have created an ic file with the dimensions of your new grid: cami-mam4_0000-01-01_ne0np4.Nanjing.ne30x8_L32_c240809.nc

On your own computer, you can find ${VRM_tools}/gen_CAMncdata/TEMPLATES/interpic_script_TEMPLATE.sh, and use the interpic executable that you created. You can start with any *.cam.i.* file that you have available.

Regrid 'atmsrf' file

The resulting file is assigned to drydep_srf_file in user_nl_cam.

On casper: copy the template script to your working directory and edit for your grid. Be sure directory $REPO/maps exists.

> cd $REPO/atmsrf
> cp /glade/work/emmons/tutorial_Nanjing/VRM_tools/gen_atmsrf/TEMPLATES/gen_atmsrf_TEMPLATE.ncl gen_atmsrf_Nanjing.ncl
> vi gen_atmsrf_Nanjing.ncl
> module load ncl
> module load esmf

> qcmd -- 'ncl gen_atmsrf_Nanjing.ncl > log_Nanjing'

Check there are no errors in the log file. This writes atmsrf_ne0np4.Nanjing.ne30x8_240809.nc.

On your own computer: find ${VRM_tools}/gen_atmsrf/TEMPLATES/gen_atmsrf_TEMPLATE.ncl

Create Topography file

The resulting file is assigned to bnd_topo in user_nl_cam.

The following steps get the Topo software (https://github.com/NCAR/Topo), build the cube_to_target executable and run it. Documentation for Topo is available on its wiki page.

On casper:

> cd $REPO
> git clone https://github.com/NCAR/Topo.git Topo
> cd Topo
> cd cube_to_target
> module load gcc
> module list

Currently Loaded Modules:

1) ncarenv/23.10 (S) 2) ncl/6.6.2 3) gcc/12.2.0 4) ncarcompilers/1.0.0 5) hdf5/1.12.2 6) netcdf/4.9.2 7) cuda/12.2.1 8) ucx/1.14.1 9) openmpi/4.1.6 10) esmf/8.5.0

> gmake -f Makefile clean
> gmake -f Makefile

Put following in a script or be sure entire command is on one line:

> qcmd -l walltime=12:00:00 -l select=1:ncpus=1 -- ./cube_to_target --rrfac_manipulation
--grid_descriptor_file='/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_np4_SCRIP.nc'
--intermediate_cs_name='/glade/campaign/cgd/amp/pel/topo/cubedata/gmted2010_modis_bedmachine-ncube3000-220518.nc'
--output_grid='Nanjing_ne30x8' --rrfac_max=8 --smoothing_scale=100.0 -u 'Louisa Emmons, emmons@ucar.edu' > out_Nanjing

Modify the bold text to point to your SCRIP file, grid name, etc.

Can add option --output_data_directory to specify location of final topography file (be sure the directory already exists).

This can take several hours.

Generate CTSM (CLM) surface datasets

On derecho, use CTSM5.2 source code to create the needed surfdata and landuse_timeseries files.

> git clone https://github.com/ESCOMP/CTSM ctsm5.2.007
> cd ctsm5.2.007
> git checkout ctsm5.2.007
> bin/git-fleximod update

Build executable for mksurfdata

> cd ./tools/mksurfdata_esmf
> ./gen_mksurfdata_build

Generate namelist files containing specifications for creating the surface datasets.

First load the conda environment:

> module load conda/latest
> conda activate npl

Run the make namelist script (modify the bold text for your resolution, MESH file and ncol for model-mesh-nx)

(npl) > /glade/work/emmons/cesm_src_derecho/ctsm5.2.007/tools/mksurfdata_esmf/gen_mksurfdata_namelist --res Nanjing_ne30x8 --start-year 1979 --end-year 2026 --ssp-rcp SSP3-7.0 --model-mesh /glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_np4_MESH.nc --model-mesh-nx 60482 --model-mesh-ny 1

This creates files like: landuse_timeseries_SSP3-7.0_1979-2026_78pfts.txt and surfdata_Nanjing_ne30x8_SSP3-7.0_1979_78pfts_c240809.namelist.

Now create a job script (use your namelist filename):

> /glade/work/emmons/cesm_src_derecho/ctsm5.2.007/tools/mksurfdata_esmf/gen_mksurfdata_jobscript_single --number-of-nodes 4 --tasks-per-node 128 --namelist-file surfdata_Nanjing_ne30x8_SSP3-7.0_1979_78pfts_c240809.namelist

This creates mksurfdata_jobscript_single.sh. Edit this file to use your project # in '#PBS -A P...'.

> qsub mksurfdata_jobscript_single.sh

This creates the netcdf files that will be used in your simulation, specified in user_nl_clm: fsurdat = '/repo_path/land/surfdata_ ... .nc' and flanduse_timeseries = '/repo_path/land/landuse.timeseries_ ... .nc'. In your $REPO, create a directory 'land' (mkdir $REPO/land) and copy these 2 nc files to it.

> conda deactivate

Anchor
Add grid to CESM
Add grid to CESM
Set up CESM3 with new grid

On derecho, checkout a CESM3 beta tag (that uses ctsm5.2). It is recommended the source code is cloned into your work (or scratch, if necessary; you might run out of space in home) directory where sufficient storage is available. CESM3 has replaced manage_externals with git-fleximod, which requires the following commands:

> git clone https://github.com/ESCOMP/CESM.git cesm3_0_beta01
> cd cesm3_0_beta01
> git checkout cesm3_0_beta01
> ./bin/git-fleximod update

Below, $CESMCODE refers to yourcesm3_0_beta01 directory.

CCS_CONFIG

1. Add an entry for your new ESMF mesh file in <cesm>/ccs_config/component_grids_nuopc.xml

For this grid:

<domain name="ne0np4.Nanjing.ne30x8">
<nx>60482</nx> <ny>1</ny>
<mesh>/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_np4_MESH.nc</mesh>
<desc>ne0np4.Nanjing.ne30x8 is a Spectral Elem 1-deg grid with a 1/8 deg refined region over east China:</desc>
<support>Test support only</support>
</domain>

2. Add a grid alias entry in <cesm>/ccs_config/modelgrid_aliases_nuopc.xml

<model_grid alias="ne0Nanjingne30x8_ne0Nanjingne30x8_mt12" not_compset="_POP">
<grid name="atm">ne0np4.Nanjing.ne30x8</grid>
<grid name="lnd">ne0np4.Nanjing.ne30x8</grid>
<grid name="ocnice">ne0np4.Nanjing.ne30x8</grid>
<mask>tx0.1v3</mask>
</model_grid>

Note the “grid name” must match the ”domain name” in component_grids_nuopc.xml.

Anchor
Run CAM
Run CAM
Set up a CAM case for testing new grid

On derecho, go to the directory (or create one) for all your CESM cases (in your /glade/u/home/$USER or /glade/work/$USER directory). Create a new case with FHIST compset:

> /glade/work/emmons/cesm_src_derecho/cesm3_0_beta01/cime/scripts/create_newcase --case /glade/work/emmons/tutorial_Nanjing/cases/f.e3beta01.FHIST.Nanjing_ne30x8.01 --res ne0Nanjingne30x8_ne0Nanjingne30x8_mt12 --compset FHIST --run-unsupported --project AACD0004 --pecount 2048

This creates a new directory, $CASEROOT = /glade/work/emmons/tutorial_Nanjing/cases/f.e3beta01.FHIST.Nanjing_ne30x8.01.

> cd $CASEROOT
> ./case.setup

Add to user_nl_cam:

ncdata = '

...

/glade/work/emmons/tutorial_Nanjing/

...

Edit input-Nanjing_ne30x8.nl to have your path to $REPO/grids and the grid name. Then run Gen_ControlVolumes:

...

ne0np4.Nanjing.ne30x8/cami-mam4_0000-01-01_ne0np4.Nanjing.ne30x8_L32_c240809.nc'
bnd_topo = '/glade/work/emmons/tutorial_Nanjing/

...

This produces Nanjing_ne30x8_np4_SCRIP.nc and Nanjing_ne30x8_np4_LATLON.nc

Examine the LATLON file to get the number of columns in your new grid:

> ncdump -h Nanjing_ne30x8_np4_LATLON.nc

This prints (among other things):

ncol = 60482 ;

Create ESMF mesh file from SCRIP file

On casper, load needed modules:

> module load mpi-serial/2.3.0
> module load esmf/8.5.0

...

> module show esmf/8.5.0

Find the listing for the "PATH" directory:

"PATH","/glade/u/apps/casper/23.10/spack/opt/spack/esmf/8.5.0/mpi-serial/2.3.0/oneapi/2023.2.1/dfkx/bin"

In the directory with your SCRIP file, run:

> /glade/u/apps/casper/23.10/spack/opt/spack/esmf/8.5.0/mpi-serial/2.3.0/oneapi/2023.2.1/dfkx/bin/ESMF_Scrip2Unstruct Nanjing_ne30x8_np4_SCRIP.nc Nanjing_ne30x8_np4_MESH.nc 0

Note the 0 at the end.

This creates the ESMF mesh file: Nanjing_ne30x8_np4_MESH.nc

Generate CESM input files on new grid

To keep with previous conventions for the structure of your new grid repository create directories inic, atmsrf and topo in your repository:

> cd $REPO
> mkdir inic
> mkdir maps
> mkdir atmsrf
> mkdir topo

The files created below are generally labeled with today's date. Even if you make files on different days, all the files in one repository should have the same date (YYMMDD).

Regrid CAM IC file

The resulting file is assigned to ncdata in user_nl_cam.

...

ne0np4.Nanjing.ne30x8/topo/topo/Nanjing_ne30x8_gmted2010_modis_bedmachine_nc3000_Laplace0100_noleak_20240729.nc'
drydep_srf_file = '/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/atmsrf/atmsrf_ne0np4.Nanjing.ne30x8_240809.nc'
se_refined_mesh = .true.
se_mesh_file = '/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_EXODUS.nc'

inithist = 'DAILY'

Set the timestep based on recommended value in: https://github.com/ESMCI/Community_Mesh_Generation_Toolkit/blob/master/VRM_tools/Docs/CAM-tsteps-inic-for-newgrids_v0.pdf

./xmlchangeATM_NCPL=384

Add to user_nl_clm:

fsurdat = '/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/land/surfdata_Nanjing_ne30x8_SSP3-7.0_1979_78pfts_c240809.nc'
flanduse_timeseries = '/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/land/landuse.timeseries_Nanjing_ne30x8_SSP3-7.0_1979-2026_78pfts_c240809.nc'

Add the override flag in your individual case directory to ensure CTSM doesn’t error out due to an unsupported grid:

./xmlchange --append CLM_BLDNML_OPTS="-no-chk_res"

> qcmd -- ./case.build

Set up to run a few days.

./xmlchange RUN_STARTDATE="2010-01-01"

./xmlchange STOP_N="5"

If it does not run, try adjusting parameters, as described in https://github.com/ESMCI/Community_Mesh_Generation_Toolkit/blob/master/VRM_tools/Docs/CAM-tsteps-inic-for-newgrids_v0.pdf

Once it runs, change the frequency of writing IC files from daily to monthly (in user_nl_cam: inithist = 'MONTHLY') and run a year. Save the final *cam.i.* file to use for future CAM simulations (ncdata in user_nl_cam).

Save the final CLM restart file (*.clm2.r.*.nc) to use for finidat in user_nl_clm in future runs.

Anchor
Run CAM-chem
Run CAM-chem
Set up a CAM-chem case for production

Create a case with FCnudged compset

> /glade/work/emmons/cesm_src_derecho/cesm3_0_beta01/cime/scripts/create_newcase --case /glade/work/emmons/tutorial_Nanjing/

...

cases/f.e3beta01.FCnudged.Nanjing_ne30x8.01 --res ne0Nanjingne30x8_ne0Nanjingne30x8_mt12 --compset FCnudged --run-unsupported --project AACD0004 --pecount 2048

Regrid IC file

You will need to regrid a previous CAM-chem IC file to your grid (to use for ncdata in user_nl_cam). If you have an IC file on a Finite Volume grid (f09) use 'interpic' as shown above. If starting from a file on ne30 grid, use the ncl script at: https://github.com/NCAR/IPT/tree/master/Initial_conditions

A sample script for regridding a f09 CAM-chem file to the new grid is: /glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/inic/interpic_script_camchem_Nanjing.sh

Regrid emissions files to your grid

Regridding emissions

Python regridding function: https://ncar.github.io/CAM-chem/examples/functions/Regridding.html

Example Python notebooks for regridding: /glade/u/home/emmons/EMISSIONS/regrid_notebooks_samples/

Regrid meteorology (MERRA2 or GEOS-FP) files to your grid

Regridding meteorological data

Update user_nl_clm

Use the CLM restart file from your CAM spinup for finidat in user_nl_clm.

Update user_nl_cam

If simulating post-2015, update lower boundary conditions file:

flbc_file = '/glade/p/cesmdata/cseg/inputdata/atm/waccm/lb/LBC_17500116-25001216_CMIP6_SSP585_0p5degLat_c20200824.nc'

Add list of emissions files on new grid (srf_emis_specifier and ext_frc_specifier)

Update sea surface temperature (SST) file

./xmlchange SSTICE_DATA_FILENAME='/glade/p/cesmdata/inputdata/atm/cam/sst/sst_HadOIBl_bc_1x1_1850_2021_c141021.nc'
./xmlchange SSTICE_YEAR_END=2021

Using existing Nanjing_ne30x8 grid files

After creating a new case and running ./case.setup, do the following steps:

Copy contents of

Edit the script to point to your grid files.

# USER CHANGES
VRdate="YYMMDD"
VRgridName="ne0np4.NAME.ne30xR"
VRgridLabel="NAME_RESOL"
VRrepoPath="your_repo_path"
# end of USER CHANGES

On casper:

> module load nco
> module load ncl
> qcmd -- 'sh interpic_script_Nanjing.sh > log_Nanjing'

Check there are no errors in the log file. This should have created an ic file with the dimensions of your new grid: cami-mam4_0000-01-01_ne0np4.Nanjing.ne30x8_L32_c240809.nc

Regrid 'atmsrf' file

The resulting file is assigned to drydep_srf_file in user_nl_cam.

Copy the template script to your working directory and edit for your grid. Be sure directory $REPO/maps exists.

> cd $REPO/atmsrf
> cp /glade/work/emmons/tutorial_Nanjing/VRM_tools/gen_atmsrf/TEMPLATES/gen_atmsrf_TEMPLATE.ncl gen_atmsrf_Nanjing.ncl
> vi gen_atmsrf_Nanjing.ncl
> module load ncl
> module load esmf

> qcmd -- 'ncl gen_atmsrf_Nanjing.ncl > log_Nanjing'

Check there are no errors in the log file. This writes atmsrf_ne0np4.Nanjing.ne30x8_240809.nc.

Create Topography file

The resulting file is assigned to bnd_topo in user_nl_cam.

...

/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/

...

user_

...

nl_

...

cam to user_nl_cam in your case directory
Copy contents of /glade/

...

work/

...

emmons/

...

tutorial_Nanjing/ne0np4.Nanjing.ne30x8/user_nl_clm to user_nl_clm in your case directory
In your case directory, run xmlchange commands listed in /glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/shell_commands

Then build your case (>qcmd -- ./case.build )

...

Set up CESM3 with new grid

Set up a CAM case for testing new grid

...

Child pages

Page History

Versions Compared

Old Version 16

New Version Current

Key

AnchorIntroIntroNaming Conventions

Set up a repository for your grid

AnchorCreate GridCreate GridCreate a new variable resolution mesh (VRM)

Get and build VRM Tools: VRM_Editor and Create_VRMgrid

Set up the base grid

Create draft refinement region

Start VRM_Editor

Set up base grid

Create grid files

Create SCRIP and LATLON grid files from EXODUS file

Create draft refinement region

AnchorCreate mesh filesCreate mesh filesCreate grid files

Create SCRIP and LATLON grid files from EXODUS file

Create ESMF mesh file from SCRIP file

AnchorInput filesInput filesGenerate CESM input files on new grid

Regrid CAM IC file

Regrid 'atmsrf' file

Create Topography file

Generate CTSM (CLM) surface datasets

AnchorAdd grid to CESMAdd grid to CESMSet up CESM3 with new grid

CCS_CONFIG

AnchorRun CAMRun CAMSet up a CAM case for testing new grid

Create ESMF mesh file from SCRIP file

Generate CESM input files on new grid

Regrid CAM IC file

AnchorRun CAM-chemRun CAM-chemSet up a CAM-chem case for production

Create a case with FCnudged compset

Regrid IC file

Regrid emissions files to your grid

Regrid meteorology (MERRA2 or GEOS-FP) files to your grid

Update user_nl_clm

Update user_nl_cam

Update sea surface temperature (SST) file

Using existing Nanjing_ne30x8 grid files

Regrid 'atmsrf' file

Create Topography file

Set up CESM3 with new grid

Set up a CAM case for testing new grid

Anchor
Intro
Intro
Naming Conventions

Anchor
Create Grid
Create Grid
Create a new variable resolution mesh (VRM)

Anchor
Create mesh files
Create mesh files
Create grid files

Anchor
Input files
Input files
Generate CESM input files on new grid

Anchor
Add grid to CESM
Add grid to CESM
Set up CESM3 with new grid

Anchor
Run CAM
Run CAM
Set up a CAM case for testing new grid

Anchor
Run CAM-chem
Run CAM-chem
Set up a CAM-chem case for production