The plot of GPSdiff_max for May 22, 2011 shows a maximum value of around 25 seconds between 11:05 and 17:00 UTC, indicating something went haywire with the data collection during those times.
Looking at the downloaded data, it appears that data stopped coming in on the Diamond serial ports at 11:05, and resumed at 17:00. There is not a large gap in data from the 3 sensors that are sampled on the Viper serial ports, but they show large time tag delta-Ts, followed by many small delta-Ts, symptomatic of the system getting behind and having to do major buffering. Perhaps a "storm" of interrupts from the Diamond that finally cleared up? Moisture somewhere?
I don't have time to look into it at this time, but here are some dumps of data around the problem.
data_stats /scr/isfs/projects/BEACHON_SRM/raw_data/manitou_20110522_000000.dat
2011-05-24,16:56:30|INFO|opening: /scr/isfs/projects/BEACHON_SRM/raw_data/manitou_20110522_000000.dat
EOFException: /scr/isfs/projects/BEACHON_SRM/raw_data/manitou_20110522_000000.dat: open: EOF
sensor dsm sampid nsamps |------- start -------| |------ end -----| rate minMaxDT(sec) minMaxLen
1 20 39901 2011 05 22 00:00:00.998 05 22 11:05:01.461 1.00 0.336 1.643 19 19
1 30 84632 2011 05 22 00:00:00.571 05 22 11:59:59.719 1.96 0.042 26.076 51 73
1 100 843525 2011 05 22 00:00:00.160 05 22 11:59:59.944 19.53 -2.052 27.951 12 16
1 110 415176 2011 05 22 00:00:00.113 05 22 11:59:59.959 9.61 -2.016 28.058 39 87
1 120 41042 2011 05 22 00:00:00.185 05 22 11:05:01.388 1.03 0.294 1.643 29 31
1 200 798028 2011 05 22 00:00:00.166 05 22 11:05:01.873 20.00 0.005 0.596 12 12
1 220 40687 2011 05 22 00:00:00.791 05 22 11:05:01.158 1.02 0.357 1.604 28 29
1 300 798048 2011 05 22 00:00:00.168 05 22 11:05:01.884 20.00 0.002 0.585 12 12
1 310 399023 2011 05 22 00:00:00.000 05 22 11:05:01.841 10.00 0.042 0.454 44 49
1 320 40488 2011 05 22 00:00:00.233 05 22 11:05:01.428 1.01 0.356 1.613 28 29
1 330 7981 2011 05 21 23:59:55.213 05 22 11:04:55.250 0.20 4.415 5.586 49 60
1 400 798041 2011 05 22 00:00:00.005 05 22 11:05:01.883 20.00 0.001 0.591 12 12
1 420 40772 2011 05 22 00:00:00.328 05 22 11:05:01.192 1.02 0.344 1.614 29 30
1 500 798042 2011 05 22 00:00:00.171 05 22 11:05:01.913 20.00 0.002 0.603 12 12
1 510 399020 2011 05 22 00:00:00.007 05 22 11:05:01.814 10.00 0.037 0.436 44 49
1 520 40747 2011 05 22 00:00:00.342 05 22 11:05:01.149 1.02 0.372 1.588 27 29
[maclean@porter ~]$ data_stats /scr/isfs/projects/BEACHON_SRM/raw_data/manitou_20110522_120000.dat
2011-05-24,16:57:04|INFO|opening: /scr/isfs/projects/BEACHON_SRM/raw_data/manitou_20110522_120000.dat
EOFException: /scr/isfs/projects/BEACHON_SRM/raw_data/manitou_20110522_120000.dat: open: EOF
sensor dsm sampid nsamps |------- start -------| |------ end -----| rate minMaxDT(sec) minMaxLen
1 20 25156 2011 05 22 17:00:47.215 05 22 23:59:59.960 1.00 0.019 1.371 19 29
1 30 73288 2011 05 22 12:00:25.745 05 22 23:59:59.741 1.70 0.104 26.116 51 73
1 100 750227 2011 05 22 12:00:28.065 05 22 23:59:59.959 17.38 -2.052 27.951 12 16
1 110 337790 2011 05 22 12:00:28.120 05 22 23:59:59.922 7.82 -2.016 28.058 17 69
1 120 25895 2011 05 22 17:00:47.289 05 22 23:59:59.553 1.03 0.029 1.756 29 38
1 200 502634 2011 05 22 17:01:05.737 05 22 23:59:59.999 20.00 0.000 2.518 2 466
1 220 25715 2011 05 22 17:00:47.290 05 22 23:59:59.392 1.02 0.028 1.594 28 39
1 300 502351 2011 05 22 17:01:20.419 05 22 23:59:59.968 20.00 0.000 2.538 2 465
1 310 251531 2011 05 22 17:00:47.284 05 22 23:59:59.926 10.00 0.050 0.154 44 66
1 320 25590 2011 05 22 17:00:47.291 05 22 23:59:59.734 1.02 0.028 1.315 28 39
1 330 5032 2011 05 22 11:05:00.251 05 22 23:59:50.264 0.11 0.061 21347.049 6 61
1 400 502055 2011 05 22 17:01:35.059 05 22 23:59:59.987 20.00 0.001 2.508 3 466
1 420 25769 2011 05 22 17:00:47.292 05 22 23:59:59.864 1.02 0.029 1.523 29 37
1 500 502936 2011 05 22 17:00:45.341 05 22 23:59:59.998 19.99 0.003 4.123 12 466
1 510 251530 2011 05 22 11:05:01.916 05 22 23:59:59.963 5.41 0.027 21345.401 43 50
1 520 25730 2011 05 22 17:00:47.294 05 22 23:59:59.423 1.02 0.028 1.631 26 39
Data kept coming in on the Viper serial ports (ids=30,100,110, GPS, sonic and licor at 2m), but had time tag issues, apparently because buffers filled up. Here's a snippet of the 2m sonic data in hex, showing a 28 second time tag jump, but no skip in the sonic sequence (byte 9: f8,f9,fa,fb,fc, etc)
2011 05 22 11:05:53.8243 0.05004 12 20 06 0b 03 39 ff 6e e8 f5 0f 55 aa 2011 05 22 11:05:53.8743 0.04997 12 6a 06 0f 03 1b ff 6a e8 f6 0f 55 aa 2011 05 22 11:05:53.9243 0.05002 12 bf 05 b1 02 de fe 69 e8 f7 0f 55 aa 2011 05 22 11:05:53.9743 0.04998 12 4d 05 cb 02 d0 fe 6e e8 f8 0f 55 aa 2011 05 22 11:06:21.9147 27.94 12 35 05 d6 02 df fe 71 e8 f9 0f 55 aa 2011 05 22 11:06:21.9272 0.0125 12 03 06 7f 03 30 ff 6f e8 fa 0f 55 aa 2011 05 22 11:06:21.9397 0.0125 12 32 06 7f 03 98 ff 65 e8 fb 0f 55 aa 2011 05 22 11:06:21.9522 0.0125 12 46 05 59 02 35 ff 6d e8 fc 0f 55 aa 2011 05 22 11:06:21.9647 0.0125 12 83 05 1a 03 c3 fe 75 e8 fd 0f 55 aa
After the gap, the 0.0125 dTs result from unpacking multiple samples from a buffer, where the code computes a dT = 10 bits/byte / (9600 bits/sec) * 12 bytes/sample = 0.0125 sec/sample
I see no indication of a problem in the system logs, or any weird clock adjustments in the NTP loopstats,peerstats.
1 Comment
Gordon Maclean AUTHOR
Just noticed that the same symptoms occur between Jan 19 and Jan 22, 2011. The serial ports on the emerald expansion card went out and came back on their own.