sstat
The post processing of CABL data is done on the flux laptop at the BAO tower, and on porter2 at EOL.
To check the status of the various processes on either system, use the sstat command. It displays a tree of the various processes, followed by an indication of "all services seem to be running", or it will list the missing services.
On the flux laptop, the processes are running under the aster userid. On porter2 they are running under maclean. They should be started automatically at bootup.
The processes are started and monitored by systemd on flux and porter2.
If a process isn't running, viewing the system log file, /var/log/isfs/isfs.log should indicate the problem.
On porter2, the important services are:
- nc_server: the NetCDF server process that writes data received by statsproc and R to the NetCDF files
- cabl_flab_statsproc@qc_geo_notiltcor: computes statistics from the 300m tower for the qc_geo_notiltcor dataset, i.e. the files in netcdf_geo_notiltcor
- cabl_flab_statsproc@noqc_instrument: computes statistics from the 300m tower for the noqc_instrument dataset, i.e. the files in netcdf_noqc_instrument
- cabl_flab_statsproc2@qc_geo_notiltcor: computes statistics from the bao and ehs flux stations for the qc_geo_notiltcor dataset, i.e. the files in netcdf_geo_notiltcor
- cabl_flab_statsproc2@noqc_instrument: computes statistics from the bao and ehs flux stations for the noqc_instrument dataset, i.e. the files in netcdf_noqc_instrument
- rsync_flab: script that wakes up periodically and rsync's files from flux
- R_derived: runs R every 5 minutes to write derived values to the files on netcdf_geo_notiltcor
- proc_restarter: runs every 10 seconds to see if the user has requested to restart the statsproc services
sstat will also show rsync_loop and statsproc@trh_test services. Those are running in support of the CentNet project.
On flux, the services are:
- nc_server: the NetCDF server process that writes data received by statsproc and R to the NetCDF files
- dsm_server@noqc_instrument: dsm_server process that receives and archives data from the DSMs on the tower.
- statsproc@qc_geo_notiltcor: computes statistics from the 300m tower for the qc_geo_notiltcor dataset, i.e. the files in netcdf_geo_notiltcor
- statsproc@noqc_instrument: computes statistics from the 300m tower for the noqc_instrument dataset, i.e. the files in netcdf_noqc_instrument
- rsync_dsms: script that wakes up periodically and rsync's files from the local flash drives on the DSMs on the tower
- R_derived: runs R every 5 minutes to create derived values in the files on netcdf_geo_notiltcor
- ssh_tunnel: creates the ssh tunnel to FLAB
restart_statsproc
If you make a change to the XML, you will usually want to restart the statsproc processes. Only if an XML change effects the archive of the raw data do you need to restart dsm_server.
To restart the statsproc processes on flux or porter2, use the restart_statsproc command. On flux it does a systemctl --user restart on each of the statsproc services.
On porter2 the processes are running under the "maclean" login, and only that user can directly restart the services directly. As a work-around, restart_statsproc writes a string to the file $ISFF/projects/$PROJECT/ISFF/logs/statsproc_cmd.txt. The proc_restarter service checks that file, and if it contains the string "restart", does a systemctl --user restart on the statsproc services.
.