Sensors offline

I did some investigating this morning, so I thought I'd share my analysis in case it's useful.  With ncharts, I don't think it's possible to tell the difference between a sensor that is reporting all bad values or one that is actually offline and not reporting.  Since all the DSMs are streaming samples to the dsm_server  instance on ustar , it is possible to use data_stats  on ustar to see the real-time status of all the sensors, including whether the sensor is reporting at all or not.  The command below will report *****  for any sensor which is not reporting any samples.  It's like running dsp  for the entire network:

(base) [daq@ustar ~]$ data_stats -a -p -P 10
2023-08-20,08:59:52|NOTICE|parsing: /home/daq/isfs/projects/M2HATS/ISFS/config/m2hats.xml
sensor               dsm sampid    nsamps |------- start -------|  |------ end -----|    rate minMaxDT(sec) minMaxLen
Lat.t2,...             2      3        10 2023 08 20 14:59:53.000  08 20 15:00:02.000    1.00  1.000  1.000   56   56
GPSstat.t2,...         2      4        10 2023 08 20 14:59:53.000  08 20 15:00:02.000    1.00  1.000  1.000   16   16
Stratum.t2,...         2      9         2 2023 08 20 14:59:56.193  08 20 15:00:00.194    0.25  4.001  4.001    8    8
u.4m.t1,...            2     11         0 ***********************  ******************     nan    nan    nan  nan  nan
u.4m.t2,...            2     21       597 2023 08 20 14:59:52.642  08 20 15:00:02.576   60.00  0.016  0.028   64   64
T.4m.t2,...            2     23        10 2023 08 20 14:59:52.752  08 20 15:00:01.789    1.00  0.993  1.008   12   12
...

You can even grep for the offline sensors:

(base) [daq@ustar ~]$ data_stats -a -p -P 10 | egrep '\*\*\*'
2023-08-20,09:01:21|NOTICE|parsing: /home/daq/isfs/projects/M2HATS/ISFS/config/m2hats.xml
u.4m.t1,...            2     11         0 ***********************  ******************     nan    nan    nan  nan  nan
Rainr.t0,...         140   2001         0 ***********************  ******************     nan    nan    nan  nan  nan

That shows that only the 4m sonic on t1 is not reporting.  Since the Ott only reports once a minute, we'd need a longer period to see if the Ott is not reporting.

Then you can use data_stats  to go further and check if any sensors are reporting missing values or not, by including the -D  option to print the data values:

(base) [daq@ustar ~]$ data_stats -a -p -P 60 -D -F >& data_stats.20230820_1505.txt

Looking at that output for occurrences of asterisks or NaN, it indicates other problems besides the t1 sonic.  All these sonics reported some or all NaNs over the 60-second period.  If they're reporting all NaNs, then they'll look offline to ncharts even though they're reporting, except when looking at ldiag.

• t0p, t2, t5, t6, t8, t11, t12, t14, t15, t18, t20, t23, t24, t25, t29, t32, t35, t38, t41, t44, t47, t48, and almost all sonics at t0 and tt.

Since the camera on the array has water on it, I'm hoping all the sonics except for t1 will recover once they dry off.

For sonics, it is possible to plot ldiag in ncharts to see how well a sonic is doing and whether it is reporting or not.  The 5-minute average is the percentage of measurements flagged as bad over that 5-minute period.  Likewise, data_stats reports the average of ldiag over the reporting period.  So in the data_stats output above, some sonics are missing only 2% and others are 98% bad.  Unless a sonic is reporting 100% bad, it will have 5-minute values in ncharts, so that's why ldiag is necessary to see if the sonic is completely ok or not.

Here is the data_stats output generated above.